Skip to content

Latest commit

 

History

History
executable file
·
285 lines (193 loc) · 11.2 KB

README.md

File metadata and controls

executable file
·
285 lines (193 loc) · 11.2 KB
Raw

Overview: Simple Search Service

build status

Simple Search Service is an IBM Bluemix app that lets you quickly create a faceted search engine, exposing an API you can use to bring search into your own apps. The service also creates a website that lets you preview the API and test it against your own data as well as manage your data via a simple CMS.

Once deployed, use the browser to upload CSV or TSV data. Specify the fields to facet, and the service handles the rest.

How it works

The application uses these Bluemix services:

  • a Node.js runtime
  • a Cloudant database
  • a Redis in-memory database from Compose.io (Optional)

Once the data is uploaded, you can use the UI to browse and manage your data via the integrated CMS. Additionally, a CORS-enabled API endpoint is available at <your domain name>/search. The endpoint takes advantage of Cloudant's built-in integration for Lucene full-text indexing. Here's what you get:

  • fielded search - ?q=colour:black+AND+brand:fender
  • free-text search - ?q=black+fender+strat
  • pagination - ?q=black+fender+strat&bookmark=<xxx>
  • faceting

You can use this along with the rest of the API to integrate the Simple Search Service into your apps. For a full API reference, click here.

While this app is a demo to showcase how easily you can build an app on Bluemix using Node.js and Cloudant, it also provides a mature search API that scales with the addition of multiple Simple Search Service nodes. In fact, a similar architecture powers the search experience in the Bluemix services catalog.

A more detailed walkthrough of using Simple Search Service is available here.

Architecture Diagram

Architecture of Simple Search Service

Running the app on Bluemix

The fastest way to deploy this application to Bluemix is to click the Deploy to Bluemix button below.

Deploy to Bluemix

Don't have a Bluemix account? If you haven't already, you'll be prompted to sign up for a Bluemix account when you click the button. Sign up, verify your email address, then return here and click the the Deploy to Bluemix button again. Your new credentials let you deploy to the platform and also to code online with Bluemix and Git. If you have questions about working in Bluemix, find answers in the Bluemix Docs.

Running the app locally

Clone this repository then run npm install to add the Node.js libraries required to run the app.

Then create some environment variables that contain your Cloudant URL.

# Cloudant URL
export SSS_CLOUDANT_URL='https://<USERNAME>:<PASSWORD>@<HOSTNAME>'

replacing the USERNAME, PASSWORD and HOSTNAME placeholders for your own Cloudant account's details.

Then run:

node app.js

Service Registry

The Simple Search Service utilises Etcd to discover and utilise some of our other Simple Services to extend and improve the service.

Other services that are available to the Simple Search Service are:

Enabling the Service Registry

Enabling the Service Registry requires setting an environment variable, ETCD_URL. This should be the URL of your Etcd instance including any basic HTTP authentication information

export ETCD_URL='http://username:[email protected]'

If the Service Registry is enabled, any discovered services will be displayed on the Services page, with a toggle to enable or disable these services.

Once enabled these services will automatically be integrated into the Simple Search Service.

Lockdown mode

If you have uploaded your content into the Simple Search Service but now want only the /search endpoint to be available publicly, you can enable "Lockdown mode".

Simply set an environment variable called LOCKDOWN to true before running the Simple Search Service:

export LOCKDOWN=true
node app.js

or set a custom environment variable in Bluemix.

When lockdown mode is detected, all web requests will be get a 401 Unauthorised response, except for the /search endpoint which will continue to work. This prevents your data being modified until lockdown mode is switched off again, by removing the environment variable.

If you wish to get access to the Simple Search Service whilst in lockdown mode, you can enable basic HTTP authentication by setting two more environment variables:

  • SSS_LOCKDOWN_USERNAME
  • SSS_LOCKDOWN_PASSWORD

When these are set, you are able to bypass lockdown mode by providing a matching username and password. If you access the UI, your browser will prompt you for these details. If you want to access the API you can provide the username and password as part of your request:

curl -X GET 'http://<yourdomain>/row/4dac2df712704b397f1b64a1c8e25033' --user <username>:<password>

API Reference

The Simple Search Service has an API that allows you to manage your data outside of the provided UI. Use this to integrate the SImple Search Service with your applications.

Search

Search is provided by the GET /search endpoint.

Fielded Search

Search on any of the indexed fields in your dataset using fielded search.

# Return any docs where colour=black
GET /search?q=colour:black

Fielded search uses Cloudant Search.

Free-text Search

Search across all fields in your dataset using free-text search.

# Return any docs 'black' is mentioned
GET /search?q=black

Pagination

Get the next page of results using the bookmark parameter. This is provided in all results from the /search endpoint (see example responses below). Pass this in to the next search (with the same query parameters) to return the next set of results.

# Return the next set of docs where 'black' is mentioned
GET /search?q=black&bookmark=<...>

It is possible to alter the amount of results returned using the limit parameter.

# Return the next set of docs where 'black' is mentioned, 10 at a time
GET /search?q=black&bookmark=<...>&limit=10

Example Response

All searches will respond in the same way.

{
  "total_rows": 19, // The total number of rows in the dataset
  "bookmark": "g1AAAA...JjFkA0kLVvg", // bookmark, for pagination
  "rows": [  // the rows returned in this response
    { ... },
    { ... },
    { ... },
    { ... },
    { ... },
    { ... },
    { ... },
    { ... },
    { ... },
    { ... },
    { ... },
    { ... },
    { ... },
    { ... },
    { ... },
    { ... },
    { ... },
    { ... },
    { ... }
  ],
  "counts": { // counts of the fields which were selected as facets during import
    "type": {
      "Black": 19
    }
  },
  "_ts": 1467108849821
}

Get a specific row

A specific row can be returned using it's unique ID, found in the _id field of each row. This is done by using the GET /row/:id endpoint.

GET /row/44d2a49201625252a51d252824932580

This will return the JSON representation of this specific row.

Add a new row

New data can be added a row at a time using the POST /row endpoint.

Call this endpoint passing in key/value pairs that match the fields in the existing data. There are NO required fields, and all field types will be enforced. The request will fail if any fields are passed in that do not already exist in the dataset.

POST /row -d'field_1=value_1&field_n=value_n'

The _id of the new row will be auto generated and returned in the id field of the response.

{
	"ok":true,
	"id":"22a747412adab2882be7e38a1393f4f2",
	"rev":"1-8a23bfa9ee2c88f2ae8dd071d2cafd56"
}

Update an existing row

Exiting data can be updated using the PUT /row/:id endpoint.

Call this endpoint passing in key/value pairs that match the fields in the existing data - you must also include the _id parameter in the key/value pairs. There are NO required fields, and all field types will be enforced. The request will fail if any fields are passed in that do not already exist in the dataset.

Note: Any fields which are not provided at the time of an update will be removed. Even if a field is not changing, it must always be provided to preserve its value.

The response is similar to that of adding a row, although note that the revision number of the document has increased.

{
	"ok":true,
	"id":"22a747412adab2882be7e38a1393f4f2",
	"rev":"2-6281e0a21ed461659dba6a96d3931ccf"
}

Deleting a row

A specific row can be deleting using it's unique ID, found in the _id field of each row. This is done by using the DELETE /row/:id endpoint.

DELETE /row/44d2a49201625252a51d252824932580

The response is similar to that of editing a row, although again note that the revision number of the document increased once more.

{
	"ok":true,
	"id":"22a747412adab2882be7e38a1393f4f2",
	"rev":"3-37b4f5c715916bf8f90ed997d57dc437"
}

Privacy Notice

The Simple Search Service web application includes code to track deployments to Bluemix and other Cloud Foundry platforms. The following information is sent to a Deployment Tracker service on each deployment:

  • Application Name (application_name)
  • Space ID (space_id)
  • Application Version (application_version)
  • Application URIs (application_uris)

This data is collected from the VCAP_APPLICATION environment variable in IBM Bluemix and other Cloud Foundry platforms. This data is used by IBM to track metrics around deployments of sample applications to IBM Bluemix. Only deployments of sample applications that include code to ping the Deployment Tracker service will be tracked.

Disabling Deployment Tracking

For manual deploys, deployment tracking can be disabled by removing require("cf-deployment-tracker-client").track(); from the end of the app.js main server file.

License

Copyright 2016 IBM Cloud Data Services

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.