View
214
Download
0
Embed Size (px)
Citation preview
February 12, 2011
NEMO All-Hands Meeting:Database and Portal
Kurt Mueller and Jason Sydes
http://nemo.nic.uoregon.edu
Agenda
• FTP site / file database overview
• NEMO portal– Present– Future
Database – FTP site
• Place to store files– Raw waveforms (input files)– Intermediate analysis results– Parameter files– Final output files
Organization
• Organize files by lab, experiment, file type:
LAB DIRECTORYEXPERIMENT DIRECTORY
INPUT_DATA DIRECTORYANALYSIS_1 DIRECTORYANALYSIS_2 DIRECTORY
Example
JTC (lab-level folder)JTC-PMN1 (experiment-level folder)
JTC-PMN1_Input_Data (subdir)'JTC-PMN1.raw' (input data
file)JTC-PMN1_Decomposition_2011-01-15 (subdir)
'JTC-PMN1_tPCA.m' (metascript file)'JTC-PMN1_DecompObj-20110115.mat' (MATLAB
output file)'JTC-PMN1_tPCA.raw' (simple binary output
data file)
Uploading files – ftp client
Data in FTP is both managed and unmanaged
• Mixture of two different areas – Unmanaged– Portal managed
• Possible area for documentation?• What role will FTP server will serve in future?• Can browse and download portal data via FTP– Readonly!
Data in portal accessible via FTP
Graphical ftp clients
• Mac OS X + Windows + Linux– Filezilla - http://filezilla-project.org/
• Mac OS X + Windows– Cyberduck – free - http://cyberduck.ch/
• Mac OS X– Transmit - $34 - http://www.panic.com/transmit/
Goals of portal
1. Online record of experiment metadata2. Repository of experiment data files3. Tight coupling to NEMO ontology
4. Provenance of all data5. NEMO analysis pipeline execution 6. Querying on metadata
now
in progre
ss
Record experiment metadata
ExperimentConditions Sessions
Publications Subject groups Data files
Task
Stimuli
Responses
Quick Sampling of Portal
Portal database tables
Repository of experiment data files
• New upload widget provides progress feedback (standard http upload does not)
OpenID access to NEMO resources
• SSO: Single Sign On• Login once, get access to– NEMO portal– NEMO wiki– NEMO sourceforge page
• Learn about it: http://openid.net/• How to create a new SSO account:http://nemo.nic.uoregon.edu/wiki/New_Nemo_User
Role-based access control (RBAC)
Database coupled to OntologyExample Table: Stimulus
Database coupled to Ontology Example “Ontology”: Stimulus
Database coupled to Ontology
Database coupled to OntologyExample mapping: Stimulus
Database coupled to Ontology Example term: stimulus_intensity
Database coupled to OntologyExample tooltip
Database coupled to OntologyQuick syncing
Open Provenance Model(http://openprovenance.org/)
Open Provenance Model– subset used –
Open Provenance ModelExample
Open Provenance ModelSimplified Example
Workflow – MATLAB in portal
• Current workflow:1. Conduct experiment, producing raw waveforms2. Process data with NEMO Toolkit scripts, locally3. Upload raw and processed data to ftp or portal4. Create an experiment in the portal, associate
data with experiment
Worfklow – MATLAB in portal
• Future workflow (in design)1. Conduct experiment, producing raw waveforms2. Create an experiment in the portal3. Upload raw data through portal4. Execute MATLAB pipeline through portal• Automatic storage of output data• Automatic provenance tracking• Single place for MATLAB scripts to live• Run on big iron
Questions?
Thank you!