• Start to develop some new code.
• Pretty early on write some tests for this code (sometimes, but frankly rather rarely I might even start with the tests, aka test-driven development or TDD).
• Iterate:
  • Run the tests (usually that will fail first time through)
  • Fix up the code
  • Run the tests; stop when they pass.
• Go back to the code and hone the ideas (yes, I often don’t really know what I need to do until I start doing it…)
• Iterate:
  • Run the tests.
  • Fix up the code and/or the tests (at this point, I might discover the tests that I originally wrote are not the right ones).
  • Run the tests; stop when they pass.
• Etc.

This is nice. In the Python/pytest setting. The fixing up part often requires knowledge of the state of the program at various points. This is because at the point where something needs fixing, I am usually facing some kind of bug in my code, and without the ability to explore the state of the program (i.e., the values of variables within the program up and down the stack), it is sometimes hard to know what’s wrong. Pytest enables this kind of examination by using the --pdb flag. This means that when tests are run with (using a recent example):

pytest afqinsight/tests/test_parametric.py --pdb

When the program hits an error, it will drop me into good old pdb, which is the Python debugging environment. If I am getting results that don’t make sense, I will often need to introduce “break points” into the program. That is, I will want to examine the program failing at a point earlier from the point at which the error is raised. This is because, for example, by the time an assertion error is raised in my tests, the state of the program that produced the error is no longer inspectable. One sneaky way to do this is to insert a 1/0 into the code at the point at which you want to examine the state of the program. This will raise a ZeroDivisionError, at which point the program will drop me into pdb, and I can get to work (I believe I may have picked this one up from Matthew Brett, when I was wet behind the ears.) I have been doing this for so long that writing the “appropriate” code for inserting a breakpoint (import pdb; pdb.set_trace()) seems like that many too many characters. By the way, this approach works just as well if I am not doing development on the command line, but I am instead working inside of a Jupyter notebook and I hit some results that don’t make sense. This is thanks to the Jupyter debug magic command, which also drops me into the debugger at the point at which an error was raised. This is all good and simple.

However, I more recently started developing software in R (here), and I have been hankering for a similar workflow. Much of what I described above does work quite well. For example, R provides an excellent test harness in the testthat library, so I can replicate my quasi-TDD development cycle quite nicely (adding in frequent visits to R documentation, stack overflow and google to figure out how to do the most basic things in this language, because I am quite clueless about it for now.)

but when it comes to debugging, testthat seems to completely resists the kind of facilities that I have been accustomed to in my Python development. For example, there is no way (that I could figure out) to run testthat in a way that would drop you into a debugging session when an error is raised(either in the test code, or in the program that is exercised by the test), which would be akin to the --pdb flag . So, for example, if I am working in the zsh shell (because, yes, I am a cliche) and execute:

Rscript -e "devtools::test()"¹

Errors will get recorded and reported, but as far as I can tell, there is no input to the test function that will make R stop running when it hits that error and would drop me into an interactive debugging session. If I introduce breakpoints into the code by introducing browser() calls into the code, testthat will happily report that a browser debugging session happened, but will just keep going. So, how does one (especially one as inexperienced with R as me) make any progress debugging their code?

Enter Positron

So here’s what I end up doing (for now?). I use the newly-launched Positron IDE. I would’ve preferred to be in VS Code (my usual text editor) for all this, but fortunately, it’s a fork of VS Code, so I am pretty close to where I want to be. I can still use the terminal to run the tests and identify failures, repeating my development-cycle-of-choice but in addition, I also add to the top of each of my test files (haven’t done it yet, to be completely honest, but it might happen soon)

library(testthat)

and append to all the calls to library code the necessary tractable::, so that when the test file is run independently in a Positron session, the code can be run and all the functions that are defined within the tests can properly be found by R. Thus, I open up the test files and ask Positron to run them for me, and when I do that, I can now hit the calls to browser and Positron will conveniently drop me into a browser session, where I can haplessly bumble through the variable defined in my code and try to make sense of things. For now, that seems to solve the main problem for me, and I can get on causing some real damage. I will also add that the Makefile I have for this library also has a make reinstall rule, which runs:

R CMD REMOVE tractable
R CMD INSTALL .

Because I can’t figure out how to install an R package in “editable” mode (i.e., the equivalent to the Python pip install -e flag), so whenever I make any changes to the code, I can run:

make reinstall
make test

to see what I’ve broken this time.

When presenting this work at the eScience Institute core staff meeting a few months ago, Anissa Tanweer raised the interesting question of whether we could reason about the data in advance of the rather extensive empirical evaluation that we did to tell us whether this data would benefit from the additional flexibility and power of an NN algorithm. My current hunch is that this is not possible, because it’s hard to discover a specific non-linear relationship in high dimensional data without the non-linear model. However, I think that there could be a way to reason about data that would bring us closer to this interesting silver bullet. This could have to do with the fact that different types of data have different characteristics. In the paper that we are writing, we touch on the fact that tract profiles are derived from images, but they are no longer images. In fact, when they are naively observed, they are tabular data. But we also know that they have group structure, and that they contain spatially contiguous data in neighboring nodes within a tract, but not across tracts. In addition, there could be other kinds of structure. For example, some tissue properties that we calculate could be correlated because of physical relationships between them (for example, diffusion FA and estimates of axonal water fraction are related to each other). Another example is that there are known correlations between the same tract in the two hemispheres (a fact beautifully exploited by Lerma et al.).

One way to analyze and discover this structure may be through unsupervised learning approaches (linear and non-linear). If this analysis reveals the kind of structure that would benefit from the strengths of the NN, then it might be worth pursuing.

This also relates to a typology of different kinds of data that we might encounter with more or less of this kind of structure. For example, things that are more “image-like” vs. things that are more “table-like”. It also relates to findings about the relative disadvantage of CNNs in analysis of large tabular data, where other kinds of non-linear models may be better suited.

One of the things that I hope to do next is an overhaul of the documentation. I have noticed that we field a set of the same questions from collaborators and I am hopeful that rethinking the documentation will have an impact on our ability to collaborate with researchers from a variety of backgrounds who want to use the methods, and will also enable us to more effectively capture the knowledge that we accumulate through these interactions. I am excited about the framework that is laid out in divio’s documentation system, and we are currently working to implement this with Qiqi Liang, who recently joined the group as a research assistant. This is currently work in progress in a PR that will hopefully replace our current documentation before too long.

So, I set about to create an AKS cluster following the relevant Dask documentation. The first step was to use the zero to jupyterhub documentation to create an AKS cluster. The only thing that needed tweaking there was the kubernetes version used in the aks cluster create call. I ended up with 1.19.6, based mostly on the fact that I have 1.19.3 installed locally. I also set this up with a minimum of 3 instances and a max of 100. That should be enough.

The Dask documentation is not very explicit about how to use multiple files for configuration (e.g., config.yaml vs. secrets.yaml). I think that I can split this up arbitrarily, but not 100% sure how to do that, so to be on the safe side, I just kept everything together in one file called secrets.yaml. After an initial install of the helm chart following the Dask documentation, I got the IP address for the cluster, and set up GitHub authentication. After doing that, I set up a custom docker image and installed that into the hub (more about this image below). After progressively changing this file, I ended up with the following ('xxx' are my secrets):

jupyterhub:
  singleuser:
    extraEnv:
      DASK_GATEWAY__CLUSTER__OPTIONS__IMAGE: '{JUPYTER_IMAGE_SPEC}'
      DASK_DISTRIBUTED__DASHBOARD_LINK: '/user/{JUPYTERHUB_USER}/proxy/{port}/status'
      DASK_LABEXTENSION__FACTORY__MODULE: 'dask_gateway'
      DASK_LABEXTENSION__FACTORY__CLASS: 'GatewayCluster'
    image:
      name: ghcr.io/nrdg/autofq-daskhub
      tag: 0546ce5f054b

  proxy:
    secretToken: "xxx"
  hub:
    networkPolicy:
      enabled: false

    readinessProbe:
      enabled: false
    services:
      dask-gateway:
        apiToken: "xxx"
    config:
      Authenticator:
        admin_users:
        - arokem
      GitHubOAuthenticator:
        client_id: xxx
        client_secret: xxx
        oauth_callback_url: http://xxx/hub/oauth_callback
      JupyterHub:
        admin_access: true
        authenticator_class: github
  cull:
    enabled: false

dask-gateway:
  gateway:
    auth:
      jupyterhub:
        apiToken: "xxx"
    extraConfig:
      optionHandler: |
            from dask_gateway_server.options import Options, Integer, Float, String
            def option_handler(options):
                if ":" not in options.image:
                    raise ValueError("When specifying an image you must also provide a tag")
                return {
                    "image": options.image,
                }
            c.Backend.cluster_options = Options(
                String("image", default="ghcr.io/nrdg/autofq-daskhub:0546ce5f054b", label="Image"),
                handler=option_handler,
            )
  traefik:
    service:
      type: ClusterIP  # Access Dask-Gateway through JupyterHub.

A few things to note about this:

1. I am cargo-culting Erik’s work here. For example, I am not sure whether the traefik section at the end is necessary or if the readinessProbe section under jupyterhub.hub. They don’t seem to harm.
2. The networkPolicy.enabled == false bit, however, is essential for GatewayCluster to work. This is based on Erik’s work on the l2l hub, and also on this issue (I get a ServerDisconnectedError error otherwise).
3. For things to work smoothly, I had to change the machine type that Azure uses on the AKS nodepool to the beefier ‘Standard_D8s_v3’ (from Standard_D4s_v3 that was there originally, I believe). Otherwise, scaleup events (e.g., adding workers) can cause everything to come crashing down. I think it’s because scaleups require more memory than each one of these machines had.
4. I don’t like it when Jupyterhub culls pods. In particular, it seems that the criterion for culling is not whether any computing is happening on the pod, but if there is enough interaction. This can be very frustrating if you are trying to run long-running computations, while doing something else (like, say, writing this).

The ghcr.io/nrdg/autofq-daskhub image configuration is maintained on this repository.

A few important points about the Docker configuration:

1. I spent some time trying to inherit from an image higher up in the dependency chain (as suggested here), but I ended up using the Dockerfile that Erik had written for the l2l hub instead.
2. It intentionally does everything, without using any of the onbuild tricks that the pangeo stack uses. Again, this relies on Erik’s lead on the l2l image config.
3. I still have some work to do to figure out how to install some tricky dependencies. In particular, some of our software relies on cvxpy for convex optimization methods, and that turns out to be a bit of a pain to install, even with conda (why oh why are there so many conflicts with conda? Is that a new thing?).
4. One big gotcha with ghcr is that images are per default private and you have to explicitely make an image public for the hub to be able to pull that.

The initial installation of the hub and upgrades are done via:

helm upgrade --wait --install --render-subchart-notes \
    dhub dask/daskhub \
    --version=4.5.7 \
    --values=secrets.yaml

The code I use to start a cluster and use it in notebooks:

from dask_gateway import GatewayCluster
gateway = GatewayCluster()
gateway.scale(n_workers)
client = gateway.get_client()

Things I still need to sort out:

1. This issue is still happening with this configuration. That is, using the dask labextension currently doesn’t work (and could confuse users, as it launches a LocalCluster). I can still get a view on the work that the cluster is doing by clicking on the provided dashboard link.
2. The initialization of the GatewayCluster class instance is still a bit flaky. It looks like maybe it takes a while for the dask-scheduler pod to get started on a cold cluster, so I sometimes need to repeat this step a couple of times, before it works.
3. My current configuration has 2GB RAM per worker. That might be a little low for the kinds of things we’d like to do with this.
4. It would be great to automate everything, so that changes to configuration immediately trigger an update to the hub. Right now, I have to make a PR for the docker image to build (at least that’s automated!), then merge the PR and wait for it to build, copy the hash of the new version of the image into my secrets.yaml file and run the upgrade on my laptop. I know it’s possible to have the CI/CD do all of this, but I would still need to piece it together. I realize that this will save more time in the long run. Sigh.

Finally, something a bit more meta. All this made me think of conversations that I was recently part of where people said things like “we are scientists, not cloud engineers!”, suggesting that large research collaborations should rely (exclusively?) on non-scientists with specialized knowledge to design, implement and maintain systems like this. I’d like to push against that notion just a bit. It’s true that I am relying on Erik and others with the specialized expertise as a starting point for this work, but I think that understanding how these systems and technologies work a bit better is relevant to scientific knowledge creation in many fields where this technology is going to be used. Just like I believe software engineering is. Or math. This doesn’t neccesarily mean that we should drop everything and just focus on cloud engineering issues, but we should try to make sure that it has a place within our work. For me personally, one of my main learning points from all this was to more effectively use kubectl describe pod to debug issues that come up and try to interpret events that happened that caused issues to arise. There’s definitely more to learn.

EDIT (20210215): After first writing this post, I figured out some things that were originally wrong, so I fixed them in the config file that I posted here. I also discovered (maybe that shouldn’t have been a surprise?) that some issues that come up can be “fixed” by turning the damn thing off and then on again. In this case, I ran into some really puzzling behaviors with dask schedulers dying on me left and right, leaving me unable to run anything on the cluster (basically a persistent form of issue 2 above). Power cycling the Azure VMs seems to have resolved that issue.

One of the things that I have been working in the past week or so is a new and rather exciting development, spearheaded by Ben Dichter and Daniel Sotoude. Based on previous work in the geoscience community, they have been working on enabling reading of NWB files stored as HDF5 into ZARR: https://github.com/catalystneuro/HDF5Zarr. This means that a large NWB file containing multi-channel recordings of neurophysiology can be stored in cloud storage and through a combination of gcsfs and their work read into a ZARR accessible on a jupyterhub node.

My first set of experiments with this approach uses a 70 GB neurophysiology recording, provided by Yoni Browning and Beth Buffalo. The details of their experimental setup and what exactly is in these time-series is interesting, but not so important for what follows.

The first thing that can be done in this setup is to look at the data. All of it. Even though the hub is running on a machine with only 13 GB of RAM, we can access it through ZARR/gcsfs at near-interactive speed. For example, I implemented the following widget:

chan = widgets.SelectMultiple(
    options=np.arange(arr.shape[1]),
    value=[0],
    description='Channels:',
    disabled=False
    
)

time = widgets.IntSlider(
    value=0,
    min=0,
    max=arr.shape[-1]-5000,
    step=1,
    description='Starting timepoint:',
    disabled=False,
    continuous_update=False,
    orientation='horizontal',
    readout=True,
)

time_d = widgets.IntSlider(
    value=5000,
    min=0,
    max=10000,
    step=1,
    description='Window duration',
    disabled=False,
    continuous_update=False,
    orientation='horizontal',
    readout=True,
)



def f(time, time_d=5000, chan=[0]):
    fig, ax = plt.subplots()
    for c in chan:
        if c < 124:
            ax.plot(arr[c, time:time+time_d], label=f"Channel {c}")
    ax.legend()
    ax.set_xlabel("Time (samples)")
    ax.set_ylabel("Voltage")
    plt.show()

out = widgets.interactive_output(f, {'chan': chan, 'time': time, 'time_d': time_d})

widgets.HBox([widgets.VBox([chan, time, time_d]), out])

This is not painful: I can select a channel, or even several channels and see their time-series appear within a very short time. In fact, I suspect that most of that time is Matplotlib dealing with the rendering of all this data locally. The key seems to be that data access patterns matter. So, data should be addressed along the chunks stored in the zarr. That is arr[chan, x:x+delta] is good, but arr[chan1:chan2, x:x+delta] can be disastrous. But I need to experiment a bit more.

Second, by attaching this hub to a Dask kube cluster, we can process the data. For example, if we want to do a spectral decomposition using wavelets, we can write something like:

from mne.time_frequency import tfr_array_morlet
from functools import partial
freqs = np.logspace(1, 2, 20)
my_morlet = partial(tfr_array_morlet, sfreq=fs, freqs=freqs)
morlet_arr = arr.map_blocks(my_morlet, dtype=np.complex128, new_axis=2)

This generates the Dask array that could compute the Morlet wavelet decompisition for the entire dataset, for 20 frequency bands. This would result in a rather large variable: about 0.5 TB here. But this can still be viewed at near interactive speed:

chan = widgets.Select(
    options=np.arange(arr.shape[1]),
    value=[0],
    description='Channel:',
    disabled=False
    
)

time = widgets.IntSlider(
    value=0,
    min=0,
    max=arr.shape[-1]-500,
    step=1,
    description='Starting timepoint:',
    disabled=False,
    continuous_update=False,
    orientation='horizontal',
    readout=True,
)

time_d = widgets.IntSlider(
    value=50,
    min=0,
    max=1000,
    step=1,
    description='Window duration',
    disabled=False,
    continuous_update=False,
    orientation='horizontal',
    readout=True,
)



def f(time, time_d=5000, chan=0):
    fig, ax = plt.subplots()
    result = np.abs(morlet_arr[0, chan, :, time:time+time_d].compute())
    ax.matshow(result)
    ax.legend()
    ax.set_xlabel("Time (samples)")
    ax.set_ylabel("Frequency bin")
    plt.show()

out = widgets.interactive_output(f, {'chan': chan, 'time': time, 'time_d': time_d})

widgets.HBox([widgets.VBox([chan, time, time_d]), out])

Again, this is not painful, although it can only show one channel at a time and does this with a small, maybe one second, delay.

But this doesn’t do everything we’d like to do. For example, we want to normalize to z-score in every frequency band in every channel. I think that this could be done by creating a new array:

morlet_mean = da.mean(morlet_arr, axis=-1)
morlet_std = da.std(morlet_arr, axis=-1)
zscored = (morlet_arr[:,0,:,:] - morlet_mean[:, 0, :]) / morlet[:, 0, :]

Calculating the z-scored value here is challenging, because this would require aggregating across all of the channel data. And this is where Dask should help. I haven’t done this entire computation yet, but calculating the mean for the channel is about 10 minutes of computing. We do also need to worry about the edges of each block when doing this computation, so one of the next set of experiments to do would use the Dask arr.map_overlap method and a more elaborate Morlet function that also windows each chunk to combine with neighboring chunks.

One cool potential conclusion of the work, reiterating previous results that we’ve found in at least one more case is that deep learning algorithms can be sensitive to information that is “hidden in plain sight” in features of an image that are very subtle and would be hard to extract in a top-down manner, just based on what we think that an image represents. This allows the algorithm to point to parts of the retina that would not have been considered useful for a diagnosis, and in which you might think no information should be present based on the standard analysis of the images. This is a rather interesting and important conclusion, as data-driven approaches to analysis of images becomes much more central.

Distributed software development is challenging. It is hard to figure ut who is doing what, and what the overall architecture should look like. Even if we are following a well-worn template laid out by fmriprep. We had started doing bi-weekly telecons, but we needed an opportunity to get together in person and hammer out our process and coordinate our expectations. Luckily, I had some funding available from the Moore and Sloan Data Science Environments grant here at UW eScience that I could use to support travel and accommodation for a three-day code sprint. And so, on January 13th - 15th, we all congregated in Seattle (with the exception of Jelle, who couldn’t make it). The sprint gave us just the opportunity we needed to lay the ground work for the library, in terms of development infrastructure (testing, documentation, continuous integration, etc.) and an excellent opportunity to have some in-depth discussions about the things we would like dmriprep to do for us. At the end of all this, we could even go so far as to write down a roadmap for future developments during this year. This lays the groundwork for the telecons that we will continue to have on a bi-weekly basis (and that are open to anyone to join…).

For me personally, three things stand out as highlights. The first important thing that I take from this sprint is how I might implement the philosophy of “release early and often” more seriously in my own work in other projects. For example, it took us several years to finally release a 0.1 of pyAFQ, but it really shouldn’t have. And if we adopt some of the approaches that are part of the genetic make-up of dmriprep, with its origins in fmriprep, we will be releasing more and hopefully leveraging this to make more rapid progress and detect/fix problems with the software more rapidly.

The second thing that I was excited about is an approach that Matt developed to correcting for head motion and potentially also for eddy currents. This approach has its roots (at least in my mind) in a 2012 paper by Amitay, Jones and Assaf. The idea is that we are limited in how we can register different volumes to each other because differences between volumes are due to both artifactual effects that we’d like to correct for: motion and eddy currents; but also due to systematic effects that we’d like to retain. Different parts of the tissue lose signal because of the different gradients applied in different scans. This is particularly pernicious for high b-values (where a lot of the signal is lost) and for parts of the brain in which orientation changes gradually. The approach proposed by Amitay et al. is that a model of the diffusion in high b-values could be used to predict what the image should look like. This prediction is then used as a target for registration. This approach was subsequently popularized by Andersson and Sotiropolous in FSL’s popular eddy tool (and their approach is also described in a paper). Their model of diffusion is slightly more complex than the CHARMED model used by Amitay et al., but it’s not clear that it more accurately represents the data. Matt has really run with this approach by using the well-motivated 3D SHORE model to fit and predict the data using a cross-validation approach (he calls it SHORELine). However, in line with the goals of qsiprep, this approach would be limited to multi-shell diffusion. For dmriprep, we need to expand this approach slightly to also work for single-shell data. So, we need a model that accurately predicts the data. Matt’s previous experiments suggest that DTI systematically fails in some places (this is well-understood as a consequence of complex fiber configurations that are not well-captured by DTI). On the other hand, CSD seems to overfit. Luckily, during my postdoc, I developed a model that does exactly that: fits the data and predicts it really accurately, and fortunately this model is implemented in DIPY, including both fitting and prediction. One of the next stages in development (already prototyped by Derek here) uses this Sparse Fascicle Model as the predictive model in the heart of a SHORELine-like algorithm. To be continued!

Finally, the last take-home is my optimism about community-led projects that pool knoweldge, talent and resources across disparate groups and institutions. Working together towards shared goals, when possible, makes a lot of sense. The potential to save duplicated effort and to produce outcomes that take into account more different use-cases is tantalizing.The challenges of bridging between different work cultures, different scientific goals and inclinations, as well as between the incentive structure governing the contributions of individuals are non-trivial, but learning more about the patterns of collaboration that facilitate productive and happy collaborations is a worthwhile endeavour in and of itself. Maybe, like families, all happy collaborations are alike in some essential way? Hopefully, that’s exactly where dmriprep is headed.