Product search engine doest not take parameters.SEARCH_CATALOG_ID into consideration. Product search engine takes search category into the consideration. I have given the example to show how to set up search category.
After researching, It seems that data of search category for some of the catalogs is not present in the Demo data. If you select the catalog which does not have search category and then you try to search products in that catalog, product search engine does not get any products because search category is not available, so system will try to find the products in other catalogs.
Lets take an example of product 'pizza' which is associated with DemoCatalog. We can search it in this catalog as this particular catalog is having its search category. Now if we try to search this product in TestCatalog, we can find it there also. The reason behind this is TestCatalog is not having any specific search category and system will try to find product in all the catalogs and it found it in DemoCatalog.
Here is the example data for creating search category and product in TestCatalog :
<ProductCategory categoryName="Test Default Search" longDescription="Test Catalog Search Products - only products in this category will show up in a search in test catalog" productCategoryId="TEST_CATALOG_SEARCH" productCategoryTypeId="SEARCH_CATEGORY"/>
<ProdCatalogCategory fromDate="2015-01-17 00:00:00.0" prodCatalogCategoryTypeId="PCCT_SEARCH" prodCatalogId="TestCatalog" productCategoryId="TEST_CATALOG_SEARCH" sequenceNum="3"/>
<ProductCategoryMember fromDate="2015-01-17 00:00:00.0" productCategoryId="TEST_CATALOG_SEARCH" productId="PIZZA"/>
Also expire relation of PIZZA with DemoCatalog in ProductCateogryMember record. Now 'PIZZA' will be available only in TestCatalog and not in DemoCatalog.
Code can be fixed so that if products are not found in selected catalog, then don't show products of other catalog.