Full table of contents¶

What is CKAN?¶

CKAN is a tool for making open data websites. (Think of a content management system like WordPress - but for data, instead of pages and blog posts.) It helps you manage and publish collections of data. It is used by national and local governments, research institutions, and other organizations who collect a lot of data.

Once your data is published, users can use its faceted search features to browse and find the data they need, and preview it using maps, graphs and tables - whether they are developers, journalists, researchers, NGOs, citizens, or even your own staff.

Using CKAN¶

Registering and logging in¶

Note

Registration is needed for most publishing features and for personalization features, such as “following” datasets. It is not needed to search for and download data.

To create a user ID, use the “Login” link at the top of any page. CKAN will forward you to SOI login page. You will be prompted to select either Public or Employee account. Select and enter the information:

• Username – This should be either illinois.gov or public.external.illinois.gov account. For example, “illinois\testuser” or “central\testuser (if you are part of trusted domain)”.
• Password – enter the same password in both boxes

If there are problems with any of the fields, service will tell you the problem and enable you to correct it. When the fields are filled in correctly, CKAN will create your user account and/or automatically log you in. If you do not have an account please select to create a new one and follow the prompts.

Note

Please note you will need get in touch with State Data Practice @ “DoIT.SDP.Support@illinois.gov” to request organization administration and data publishing roles. Please submit ESR to DoIT helpdesk together with below information:

• Username
• Organization Name
• Access Type
• Justification

Features for publishers¶

Adding a new dataset¶

Note

You will need to be a member of an organization in order to add and edit datsets. On http://demo.ckan.org, you can add a dataset without being in an organization, but dataset features relating to authorization and organizations will not be available.

Step 1. You can access CKAN’s “Create dataset” screen in two ways.

1. Select the “Datasets” link at the top of any page. From this, above the search box, select the “Add Dataset” button.
2. Alternatively, select the “organizations” link at the top of a page. Now select the page for the organization that should own your new dataset. Provided that you are a member of this organization, you can now select the “Add Dataset” button above the search box.

Step 2. CKAN will ask for the following information about your data. (The actual data will be added in step 4.)

• Title – this title will be unique across CKAN, so make it brief but specific. E.g. “UK population density by region” is better than “Population figures”.
• Description – You can add a longer description of the dataset here, including information such as where the data is from and any information that people will need to know when using the data.
• Tags – here you may add tags that will help people find the data and link it with other related data. Examples could be “population”, “crime”, “East Anglia”. Hit the <return> key between tags. If you enter a tag wrongly, you can use its delete button to remove it before saving the dataset.
• License – it is important to include license information so that people know how they can use the data. This field should be a drop-down box. If you need to use a license not on the list, contact your site administrator.
• Organization - If you are a member of any organizations, this drop-down will enable you to choose which one should own the dataset. Ensure the default chosen is the correct one before you proceed. (Probably most users will be in only one organization. If this is you, CKAN will have chosen your organization by default and you need not do anything.)

Note

By default, the only required field on this page is the title. However, it is good practice to include, at the minimum, a short description and, if possible, the license information. You should ensure that you choose the correct organization for the dataset, since at present, this cannot be changed later. You can edit or add to the other fields later.

Step 3. When you have filled in the information on this page, select the “Next: Add Data” button. (Alternatively select “Cancel” to discard the information filled in.)

Step 4. CKAN will display the “Add data” screen.

This is where you will add one or more “resources” which contain the data for this dataset. Choose a file or link for your data resource and select the appropriate choice at the top of the screen:

• If you are giving CKAN a link to the data, like http://example.com/mydata.csv, then select “Link to a file” or “Link to an API”. (If you don’t know what an API is, you don’t need to worry about this option - select “Link to a file”.)
• If the data to be added to CKAN is in a file on your computer, select “Upload a file”. CKAN will give you a file browser to select it.

Step 5. Add the other information on the page. CKAN does not require this information, but it is good practice to add it:

• Name – a name for this resource, e.g. “Population density 2011, CSV”. Different resources in the dataset should have different names.
• Description – a short description of the resource.
• Format – the file format of the resource, e.g. CSV (comma-separated values), XLS, JSON, PDF, etc.

Step 6. If you have more resources (files or links) to add to the dataset, select the “Save & add another” button. When you have finished adding resources, select “Next: Additional Info”.

Step 7. CKAN displays the “Additional data” screen.

• Visibility – a Public dataset is public and can be seen by any user of the site. A Private dataset can only be seen by members of the organization owning the dataset and will not show up in searches by other users.
• Author – The name of the person or organization responsible for producing the data.
• Author e-mail – an e-mail address for the author, to which queries about the data should be sent.
• Maintainer / maintainer e-mail – If necessary, details for a second person responsible for the data.
• Custom fields – If you want the dataset to have another field, you can add the field name and value here. E.g. “Year of publication”. Note that if there is an extra field that is needed for a large number of datasets, you should talk to your site administrator about changing the default schema and dataset forms.

Note

Everything on this screen is optional, but you should ensure the “Visibility” is set correctly. It is also good practice to ensure an Author is named.

Changed in version 2.2: Previous versions of CKAN used to allow adding the dataset to existing groups in this step. This was changed. To add a dataset to an existing group now, go to the “Group” tab in the Dataset’s page.

Step 8. Select the ‘Finish’ button. CKAN creates the dataset and shows you the result. You have finished!

You should be able to find your dataset by typing the title, or some relevant words from the description, into the search box on any page in your CKAN instance. For more information about finding data, see the section Finding data.

Editing a dataset¶

You can edit the dataset you have created, or any dataset owned by an organization that you are a member of. (If a dataset is not owned by any organization, then any registered user can edit it.)

1. Go to the dataset’s page. You can find it by entering the title in the search box on any page.
2. Select the “Edit” button, which you should see above the dataset title.
3. CKAN displays the “Edit dataset” screen. You can edit any of the fields (Title, Description, Dataset, etc), change the visibility (Private/Public), and add or delete tags or custom fields. For details of these fields, see Adding a new dataset.
4. When you have finished, select the “Update dataset” button to save your changes.

Adding, deleting and editing resources¶

1. Go to the dataset’s “Edit dataset” page (steps 1-2 above).
2. In the left sidebar, there are options for editing resources. You can select an existing resource (to edit or delete it), or select “Add new resource”.
3. You can edit the information about the resource or change the linked or uploaded file. For details, see steps 4-5 of “Adding a new resource”, above.
4. When you have finished editing, select the button marked “Update resource” (or “Add”, for a new resource) to save your changes. Alternatively, to delete the resource, select the “Delete resource” button.

Deleting a dataset¶

1. Go to the dataset’s “Edit dataset” page (see “Editing a dataset”, above).
2. Select the “Delete” button.
3. CKAN displays a confirmation dialog box. To complete deletion of the dataset, select “Confirm”.

Note

The “Deleted” dataset is not completely deleted. It is hidden, so it does not show up in any searches, etc. However, by visiting the URL for the dataset’s page, it can still be seen (by users with appropriate authorization), and “undeleted” if necessary. If it is important to completely delete the dataset, contact your site administrator.

Managing an organization¶

When you create an organization, CKAN automatically makes you its “Admin”. From the organization’s page you should see an “Admin” button above the search box. When you select this, CKAN displays the organization admin page. This page has two tabs:

• Info – Here you can edit the information supplied when the organization was created (title, description and image).
• Members – Here you can add, remove and change access roles for different users in the organization. Note: you will need to know their username on CKAN.

By default CKAN allows members of organizations with three roles:

• Member – can see the organization’s private datasets
• Editor – can edit and publish datasets
• Admin – can add, remove and change roles for organization members

Finding data¶

Searching the site¶

To find datasets in CKAN, type any combination of search words (e.g. “health”, “transport”, etc) in the search box on any page. CKAN displays the first page of results for your search. You can:

• View more pages of results
• Repeat the search, altering some terms
• Restrict the search to datasets with particular tags, data formats, etc using the filters in the left-hand column

If there are a large number of results, the filters can be very helpful, since you can combine filters, selectively adding and removing them, and modify and repeat the search with existing filters still in place.

If datasets are tagged by geographical area, it is also possible to run CKAN with an extension which allows searching and filtering of datasets by selecting an area on a map.

Searching within an organization¶

If you want to look for data owned by a particular organization, you can search within that organization from its home page in CKAN.

1. Select the “Organizations” link at the top of any page.
2. Select the organization you are interested in. CKAN will display your organization’s home page.
3. Type your search query in the main search box on the page.

CKAN will return search results as normal, but restricted to datasets from the organization.

If the organization is of interest, you can opt to be notified of changes to it (such as new datasets and modifications to datasets) by using the “Follow” button on the organization page. You must have a user account and be logged in to use this feature.

Exploring datasets¶

When you have found a dataset you are interested and selected it, CKAN will display the dataset page. This includes

• The name, description, and other information about the dataset
• Links to and brief descriptions of each of the resources

The resource descriptions link to a dedicated page for each resource. This resource page includes information about the resource, and enables it to be downloaded. Many types of resource can also be previewed directly on the resource page. .CSV and .XLS spreadsheets are previewed in a grid view, with map and graph views also available if the data is suitable. The resource page will also preview resources if they are common image types, PDF, or HTML.

The dataset page also has two other tabs:

• Activity stream – see the history of recent changes to the dataset
• Related items – see any links to web pages related to this dataset, or add your own links.

If the dataset is of interest, you can opt to be notified of changes to it by using the “Follow” button on the dataset page. See the section Managing your news feed below. You must have a user account and be logged in to use this feature.

Making a DataStore API request¶

Making a DataStore API request is the same as making an Action API request: you post a JSON dictionary in an HTTP POST request to an API URL, and the API also returns its response in a JSON dictionary.

API reference¶

ckanext.datastore.logic.action.datastore_create(context, data_dict)¶

Adds a new table to the DataStore.

The datastore_create action allows you to post JSON data to be stored against a resource. This endpoint also supports altering tables, aliases and indexes and bulk insertion. This endpoint can be called multiple times to initially insert more data, add fields, change the aliases or indexes as well as the primary keys.

To create an empty datastore resource and a CKAN resource at the same time, provide resource with a valid package_id and omit the resource_id.

If you want to create a datastore resource from the content of a file, provide resource with a valid url.

See Fields and Records for details on how to lay out records.

Parameters:

resource_id (string) – resource id that the data is going to be stored against. force (bool (optional, default: False)) – set to True to edit a read-only resource resource (dictionary) – resource dictionary that is passed to resource_create(). Use instead of resource_id (optional) aliases (list or comma separated string) – names for read only aliases of the resource. (optional) fields (list of dictionaries) – fields/columns and their extra metadata. (optional) records (list of dictionaries) – the data, eg: [{“dob”: “2005”, “some_stuff”: [“a”, “b”]}] (optional) primary_key (list or comma separated string) – fields that represent a unique key (optional) indexes (list or comma separated string) – indexes on table (optional) triggers (list of dictionaries) – trigger functions to apply to this table on update/insert. functions may be created with datastore_function_create(). eg: [ {“function”: “trigger_clean_reference”}, {“function”: “trigger_check_codes”}]

Please note that setting the aliases, indexes or primary_key replaces the exising aliases or constraints. Setting records appends the provided records to the resource.

Results:

Returns:	The newly created data object.
Return type:	dictionary

See Fields and Records for details on how to lay out records.

ckanext.datastore.logic.action.datastore_run_triggers(context, data_dict)¶

update each record with trigger

The datastore_run_triggers API action allows you to re-apply exisitng triggers to an existing DataStore resource.

Parameters:	resource_id (string) – resource id that the data is going to be stored under.

Results:

Returns:	The rowcount in the table.
Return type:	int

ckanext.datastore.logic.action.datastore_upsert(context, data_dict)¶

Updates or inserts into a table in the DataStore

The datastore_upsert API action allows you to add or edit records to an existing DataStore resource. In order for the upsert and update methods to work, a unique key has to be defined via the datastore_create action. The available methods are:

upsert

Update if record with same key already exists, otherwise insert. Requires unique key.

insert

Insert only. This method is faster that upsert, but will fail if any inserted record matches an existing one. Does not require a unique key.

update

Update only. An exception will occur if the key that should be updated does not exist. Requires unique key.

Parameters:

resource_id (string) – resource id that the data is going to be stored under. force (bool (optional, default: False)) – set to True to edit a read-only resource records (list of dictionaries) – the data, eg: [{“dob”: “2005”, “some_stuff”: [“a”,”b”]}] (optional) method (string) – the method to use to put the data into the datastore. Possible options are: upsert, insert, update (optional, default: upsert)

Results:

Returns:	The modified data object.
Return type:	dictionary

ckanext.datastore.logic.action.datastore_info(context, data_dict)¶

Returns information about the data imported, such as column names and types.

Return type:	A dictionary describing the columns and their types.
Parameters:	id (A UUID) – Id of the resource we want info about

ckanext.datastore.logic.action.datastore_delete(context, data_dict)¶

Deletes a table or a set of records from the DataStore.

Parameters:	resource_id (string) – resource id that the data will be deleted from. (optional) force (bool (optional, default: False)) – set to True to edit a read-only resource filters (dictionary) – filters to apply before deleting (eg {“name”: “fred”}). If missing delete whole table and all dependent views. (optional)

Results:

Returns:	Original filters sent.
Return type:	dictionary

ckanext.datastore.logic.action.datastore_search(context, data_dict)¶

Search a DataStore resource.

The datastore_search action allows you to search data in a resource. DataStore resources that belong to private CKAN resource can only be read by you if you have access to the CKAN resource and send the appropriate authorization.

Parameters:

resource_id (string) – id or alias of the resource to be searched against filters (dictionary) – matching conditions to select, e.g {“key1”: “a”, “key2”: “b”} (optional) q (string or dictionary) – full text query. If it’s a string, it’ll search on all fields on each row. If it’s a dictionary as {“key1”: “a”, “key2”: “b”}, it’ll search on each specific field (optional) distinct (bool) – return only distinct rows (optional, default: false) plain (bool) – treat as plain text query (optional, default: true) language (string) – language of the full text query (optional, default: english) limit (int) – maximum number of rows to return (optional, default: 100) offset (int) – offset this number of rows (optional) fields (list or comma separated string) – fields to return (optional, default: all fields in original order) sort (string) – comma separated field names with ordering e.g.: “fieldname1, fieldname2 desc” include_total (bool) – True to return total matching record count (optional, default: true) records_format (controlled list) – the format for the records return value: ‘objects’ (default) list of {fieldname1: value1, ...} dicts, ‘lists’ list of [value1, value2, ...] lists, ‘csv’ string containing comma-separated values with no header, ‘tsv’ string containing tab-separated values with no header

Setting the plain flag to false enables the entire PostgreSQL full text search query language.

A listing of all available resources can be found at the alias _table_metadata.

If you need to download the full resource, read Download resource.

Results:

The result of this action is a dictionary with the following keys:

Return type:	A dictionary with the following keys
Parameters:	fields (list of dictionaries) – fields/columns and their extra metadata offset (int) – query offset value limit (int) – query limit value filters (list of dictionaries) – query filters total (int) – number of total matching records records (depends on records_format value passed) – list of matching results

ckanext.datastore.logic.action.datastore_search_sql(context, data_dict)¶

Execute SQL queries on the DataStore.

The datastore_search_sql action allows a user to search data in a resource or connect multiple resources with join expressions. The underlying SQL engine is the PostgreSQL engine. There is an enforced timeout on SQL queries to avoid an unintended DOS. DataStore resource that belong to a private CKAN resource cannot be searched with this action. Use datastore_search() instead.

Note

This action is only available when using PostgreSQL 9.X and using a read-only user on the database. It is not available in legacy mode.

Parameters:	sql (string) – a single SQL select statement

Results:

The result of this action is a dictionary with the following keys:

Return type:	A dictionary with the following keys
Parameters:	fields (list of dictionaries) – fields/columns and their extra metadata records (list of dictionaries) – list of matching results

ckanext.datastore.logic.action.datastore_make_private(context, data_dict)¶

Deny access to the DataStore table through datastore_search_sql().

This action is called automatically when a CKAN dataset becomes private or a new DataStore table is created for a CKAN resource that belongs to a private dataset.

Parameters:	resource_id (string) – id of resource that should become private

ckanext.datastore.logic.action.datastore_make_public(context, data_dict)¶

Allow access to the DataStore table through datastore_search_sql().

This action is called automatically when a CKAN dataset becomes public.

Parameters:	resource_id (string) – if of resource that should become public

ckanext.datastore.logic.action.set_datastore_active_flag(model, data_dict, flag)¶

Set appropriate datastore_active flag on CKAN resource.

Called after creation or deletion of DataStore table.

ckanext.datastore.logic.action.datastore_function_create(context, data_dict)¶

Create a trigger function for use with datastore_create

Parameters:	name (string) – function name or_replace (bool) – True to replace if function already exists (default: False) rettype (string) – set to ‘trigger’ (only trigger functions may be created at this time) definition (string) – PL/pgSQL function body for trigger function

ckanext.datastore.logic.action.datastore_function_delete(context, data_dict)¶

Delete a trigger function

Parameters:	name (string) – function name

{
                                                        "id":    # a string which defines the column name
                                                        "type":  # the data type for the column
                                                    }
                                                    

[
                                                        {
                                                            "id": "foo",
                                                            "type": "int4"
                                                        },
                                                        {
                                                            "id": "bar"
                                                            # type is optional
                                                        }
                                                    ]
                                                    

{
                                                        "<id>":  # data to be set
                                                        # .. more data
                                                    }
                                                    

[
                                                        {
                                                            "foo": 100,
                                                            "bar": "Here's some text"
                                                        },
                                                        {
                                                            "foo": 42
                                                        }
                                                    ]
                                                    

Extending DataStore¶

Starting from CKAN version 2.7, backend used in DataStore can be replaced with custom one. For this purpose, custom extension must implement ckanext.datastore.interfaces.IDatastoreBackend, which provides one method - register_backends. It should return dictonary with names of custom backends as keys and classes, that represent those backends as values. Each class supposed to be inherited from ckanext.datastore.backend.DatastoreBackend.

ckanext.datastore.backend.get_all_resources_ids_in_datastore()¶

Helper for getting id of all resources in datastore.

Uses get_all_ids of active datastore backend.

exception ckanext.datastore.backend.InvalidDataError¶

Exception that’s raised if you try to add invalid data to the datastore.

For example if you have a column with type “numeric” and then you try to add a non-numeric value like “foo” to it, this exception should be raised.

class ckanext.datastore.backend.DatastoreBackend¶

Base class for all datastore backends.

Very simple example of implementation based on SQLite can be found in ckanext.example_idatastorebackend. In order to use it, set datastore.write_url to ‘example-sqlite:////tmp/database-name-on-your-choice’

Prop _backend:	mapping(schema, class) of all registered backends
Prop _active_backend:
	current active backend

classmethod register_backends()¶

Register all backend implementations inside extensions.

classmethod set_active_backend(config)¶

Choose most suitable backend depending on configuration

Parameters:	config – configuration object
Return type:	ckan.common.CKANConfig

classmethod get_active_backend()¶

Return currently used backend

configure(config)¶

Configure backend, set inner variables, make some initial setup.

Parameters:	config – configuration object
Returns:	config
Return type:	CKANConfig

create(context, data_dict)¶

Create new resourct inside datastore.

Called by datastore_create.

Parameters:	data_dict – See ckanext.datastore.logic.action.datastore_create
Returns:	The newly created data object
Return type:	dictonary

upsert(context, data_dict)¶

Update or create resource depending on data_dict param.

Called by datastore_upsert.

Parameters:	data_dict – See ckanext.datastore.logic.action.datastore_upsert
Returns:	The modified data object
Return type:	dictonary

delete(context, data_dict)¶

Remove resource from datastore.

Called by datastore_delete.

Parameters:	data_dict – See ckanext.datastore.logic.action.datastore_delete
Returns:	Original filters sent.
Return type:	dictonary

search(context, data_dict)¶

Base search.

Called by datastore_search.

Parameters:	data_dict – See ckanext.datastore.logic.action.datastore_search fields (list of dictionaries) – fields/columns and their extra metadata offset (int) – query offset value limit (int) – query limit value filters (list of dictionaries) – query filters total (int) – number of total matching records records (list of dictionaries) – list of matching results
Return type:	dictonary with following keys

search_sql(context, data_dict)¶

Advanced search.

Called by datastore_search_sql. :param sql: a single seach statement :type sql: string

Return type:	dictonary
Parameters:	fields (list of dictionaries) – fields/columns and their extra metadata records (list of dictionaries) – list of matching results

make_private(context, data_dict)¶

Do not display resource in search results.

Called by datastore_make_private. :param resource_id: id of resource that should become private :type resource_id: string

make_public(context, data_dict)¶

Enable serch for resource.

Called by datastore_make_public. :param resource_id: id of resource that should become public :type resource_id: string

resource_exists(id)¶

Define whether resource exists in datastore.

resource_fields(id)¶

Return dictonary with resource description.

Called by datastore_info. :returns: A dictionary describing the columns and their types.

resource_info(id)¶

Return DataDictonary with resource’s info - #3414

resource_id_from_alias(alias)¶

Convert resource’s alias to real id.

Parameters:	alias (string) – resource’s alias or id
Returns:	real id of resource
Return type:	string

get_all_ids()¶

Return id of all resource registered in datastore.

Returns:	all resources ids
Return type:	list of strings

create_function(*args, **kwargs)¶

Called by datastore_function_create action.

drop_function(*args, **kwargs)¶

Called by datastore_function_delete action.

API Versions¶

There are two versions of the legacy APIs. When the API returns a reference to an object, version 1 of the API will return the name of the object (e.g. "river-pollution"), whereas version 2 will return the ID of the object (e.g. "a3dd8f64-9078-4f04-845c-e3f047125028"). Tag objects are an exception, tag names are immutable so tags are always referred to with their name.

You can specify which version of the API to use in the URL. For example, opening this URL in your web browser will list demo.ckan.org’s datasets using API version 1:

http://demo.ckan.org/api/1/rest/dataset

Opening this URL calls the same function using API version 2:

http://demo.ckan.org/api/2/rest/dataset

If no version number is given in the URL then the API defaults to version 1, so this URL will list the site’s datasets using API version 1:

http://demo.ckan.org/api/rest/dataset

Dataset names can change, so to reliably refer to the same dataset over time, you will want to use the dataset’s ID and therefore use API v2. Alternatively, many people prefer to deal with Names, so API v1 suits them.

When posting parameters with your API requests, you can refer to objects by either their name or ID, interchangeably.

Model API¶

Model resources are available at published locations. They are represented with a variety of data formats. Each resource location supports a number of methods.

The data formats of the requests and the responses are defined below.

Search API¶

Search resources are available at published locations. They are represented with a variety of data formats. Each resource location supports a number of methods.

The data formats of the requests and the responses are defined below.

curl 'http://thedatahub.org/api/search/dataset?qjson=\{"author":"The%20Stationery%20Office%20Limited"\}'
                                                                            

Util API¶

The Util API provides various utility APIs – e.g. auto-completion APIs used by front-end javascript.

All Util APIs are read-only. The response format is JSON. Javascript calls may want to use the JSONP formatting.

/api/2/util/dataset/autocomplete?incomplete=a%20novel
                                                                        

{"ResultSet": {"Result": [{"match_field": "title", "match_displayed": "A Novel By Tolstoy (annakarenina)", "name": "annakarenina", "title": "A Novel By Tolstoy"}]}}
                                                                        

/api/2/util/tag/autocomplete?incomplete=ru
                                                                        

{"ResultSet": {"Result": [{"Name": "russian"}]}}
                                                                        

/api/2/util/resource/format_autocomplete?incomplete=cs
                                                                        

{"ResultSet": {"Result": [{"Format": "csv"}]}}
                                                                        

/api/util/markdown?q=<http://ibm.com/>
                                                                        

"<p><a href="http://ibm.com/" target="_blank" rel="nofollow">http://ibm.com/</a>\n</p>"
                                                                        

/api/2/util/is_slug_valid?slug=river-quality&type=package
                                                                        

{"valid": true}
                                                                        

/api/util/dataset/munge_name?name=police%20spending%20figures%202009
                                                                        

"police-spending-figures-2009"
                                                                        

/api/util/dataset/munge_title_to_name?title=police:%20spending%20figures%202009
                                                                        

"police-spending-figures-2009"
                                                                        

/api/util/tag/munge?tag=water%20quality
                                                                        

"water-quality"
                                                                        

Status Codes¶

Standard HTTP status codes are used to signal method outcomes.