Connectomics of (part of) the MICrONS mm3 dataset

This dataset provides the connectome of a large part of the IARPA MICrONS mm^3 dataset (https://www.microns-explorer.org/cortical-mm3). Specifically, it contains internal connectivity between most neurons of "portion 65" of the EM volume (see above link for details), i.e. synapses between neurons inside the volume, but no synapses from neurons extrinsic to the volume. The volume contains parts of the regions VISp, VISrl, VISal and VISlm.

The file combines data from the following tables of the "minnie65_public_v117" release (see link above):
- allen_soma_coarse_cell_class_model_v1__minnie3_v1 for neuron identifiers, tentative classes and soma locations
- proofreading_status_public_release for information about axon / dendrite completeness
- synapses_pni_2 for synaptic connection locations, sizes, source and target neurons

The full description of those tables, as originally provided, can be found below. We converted location indicated in voxel indices in the original data to locations in nm, additionally we provide very tentative region annotations for neurons (but see below!).
The main utility of this release lies in its formatting for the Connectome-utilities python package (https://github.com/BlueBrain/ConnectomeUtilities). As such, you can easily load it and use Connectome-utilities functionality for various analyses. Exemplary notebooks are provided.

The file contains two representations of the connectome:
- "full" represent multiple synapses between neurons as multiple directed edges.
- "condensed" only has (at most) a single edge between neurons, but it is associated with a property "count" that specifies the number of synapses. Other synapse properties (such as their locations) are mostly lost in the condensed representation, only the mean and sum of the "size" property is provided. You can also get the condensed version from the full version by using the .condense() function of Connectome-utilities (see documentation).

Some notes:
- The data in the "allen soma coarse cell class model" contains some neuron identifiers associated with multiple types. Manual inspection of their meshes indicated that they really are merges of several neurons. Since this only seemed to affect a few hundred of neurons, we simply filtered them out for this dataset.
- Neurons are annotated with brain region names ("visp", "visrl", etc.). Do NOT treat them as accurate. They were derived as follows: An image depicting the EM volume with region borders drawn is provided on the MICrONS page. We aligned the x and z coordinates of neuron with the image coordinates and assigned them regions based on which side of the border they landed. This is not overly accurate and assumes region borders are parallel to the y-axis (however, note that the y-axis is very much aligned with the depth-axis). If you can provide a transformation from the original voxel coordinates to the Allen CCF, please let me know and I will use that instead.

Getting started:

To start, check the documentation of Connectome-utilities or just dive into the included exemplary jupyter notebooks.

Contact:

If you have questions or notes: conntility.645co@simplelogin.com

CREDIT
All of this is based on MICrONS, with very little work by me. To give full credit, I will include below the original description of the datasets used. Many thanks to everyone mentioned below and everyone else that worked hard to provide that highly valuable data!

allen_soma_coarse_cell_class_model_v1__minnie3_v1:
This is a model developed by Leila Elabbady and Forrest Collman, it uses features extracted from the somatic region and nucleus segmentation (developed in collaboration with Shang Mu and Gayathri Mahalingam).  Those features included the number of soma synapses, the somatic area, the somatic area to volume ratio, the density of somatic synapses, teh volume of the soma, the depth in cortex of the cell (based upon the y coordinate after a 5 degree rotation), the nucleus area, the ratio of the nucleus area to nucleus volume, the average diameter of the proximal processes of the cell,  the area of the nucleus with a fold, the fraction of the nucleus area within a fold, the volume of the nucleus, and the ratio of the volume of the nucleus to the volume of the soma.  The model was trained using labels from the allen_v1_column_types_v2 table supplemented with NP labels from allen_minnie_extra_types as of version 91 .  The model is a SVM classifier, using an rbf kernel, with class balance. On 20% of the data held out the model has a 77% accuracy, with principal confusion between layer 6 IT and CT, and layer 5 IT with layer 4 and many types. Please contact Forrest Collman for more information or questions about the model.

proofreading_status_public_release:
The proofreading status of neurons that have been comprehensively proofread within this version. Axon and dendrite compartment status are marked separately under 'axon_status' and 'dendrite_status', as proofreading effort was applied differently to the different compartments in some cells. There are three possible status values for each compartment: 'non' indicates no comprehensive proofreading. 'clean' indicates that all false merges have been removed, but all tips have not necessarily been followed. 'extended' indicates that the cell is both clean and all tips have been followed as far as a proofreader was able to. The 'pt_position' is at a cell body or similar core position for the cell. The column 'valid_id' provides the root id when the proofreading was last checked. If the current root id in 'pt_root_id' is not the same as 'valid_id', there is no guarantee that the proofreading status is correct. Very small false axon merges (axon fragments approximately 5 microns or less in length) were considered acceptable for clean neurites. Note that this table does not list all edited cells, but only those with comprehensive effort toward the status mentioned here. Table compiled by Sven Dorkenwald and Casey Schneider-Mizell, including work by many proofreaders and data maintained by Stelios Papadopoulos.

synapses_pni_2:
Automated synapse detection performed by Nick Turner from the Seung Lab.  size represents the number of (4x4x40 nm) voxels painted by the automated cleft segmentation, and the IDs reference the IDs of the cleft segmentation.  Ctr_pt reflects the centroid of the cleft segmentation. The cleft segmentation volume is located in the flat_segmentation_source field.