Rosalyn Metz (she/her): And
Rosalyn Metz (she/her): so before we get started
Rosalyn Metz (she/her): can I get a volunteer to help with taking
Rosalyn Metz (she/her): anybody?
Okay.
Andrew Woods: Andrew? All right. I'll do it. I'll do it.
Rosalyn Metz (she/her): you unmuted man.
Rosalyn Metz (she/her): That's what happens when you unmute
Andrew Woods: That's why we volunteer
Rosalyn Metz (she/her): so let's
Rosalyn Metz (she/her): that comes with saying, Yes, it totally does. if we can, because I there are a couple of
Rosalyn Metz (she/her): new faces.
Rosalyn Metz (she/her): They're not really new. But
Rosalyn Metz (she/her): I thought maybe we could quickly go around the room
Rosalyn Metz (she/her): and intro each other, or do like a brief introduction of who we are and where we're from.
Rosalyn Metz (she/her): I will start. My name is Rosalind. I am at Emory in Atlanta, and I
Rosalyn Metz (she/her): and one of the Ocfl editors
Rosalyn Metz (she/her): Daniel S. Could you introduce yourself?
Daniel Salwerowicz: Yeah, my name is Daniel Salvador, which I'm. Working for the National Library of Norway
Daniel Salwerowicz: in the digital preservation team.
Rosalyn Metz (she/her): Thank you.
Rosalyn Metz (she/her): Dan Field.
Dan Field - Fedora: Hi! Dan Field. I'm currently here wearing my Fedora hats recently started as a part-time Fedora developer with the lyricists
Dan Field - Fedora: but I also am the head of software engineering at the National Library of Wales, where we are currently undergoing a Fedora, 3 to 6 migration and using the Ocf as the underlying file system.
Rosalyn Metz (she/her): Aaron, do you want to go?
Arran Griffith - Fedora: Yeah, sure I'm Aaron Griffith, a program manager for Fedora and i'm based in Canada. So Atlantic, Canada. It's cold here, and the
Rosalyn Metz (she/her): Seth where it's less cold.
Seth Erickson: Yes, but rainy. hi I'm Saint Erickson. I'm a data services librarian at Ucsb in Santa Barbara.
Seth Erickson: and
Seth Erickson: yeah, I've been working on a implementation and go for a little while. And today I was going to share some stuff on an indexing project that I've been working on.
Rosalyn Metz (she/her): I will say that's why everybody is here today, so I hope to. hope you're not disappointed.
Rosalyn Metz (she/her): Andrew. Do you want to introduce yourself?
Andrew Woods: yes, I was hoping you would ask i'm
Andrew Woods: Andrew Woods one of the Sfl. Editors and I
Andrew Woods: manage a software engineering team at Harvard Library.
Rosalyn Metz (she/her): Doran, do you want to go.
Doron Shalvi: Sure I'm Doran Shelby. I'm. A system engineer at the Us. National Library Medicine just outside Washington, DC. And we're we're sort of in a similar situation to Dan Fields. We're long time for our 3 users, and have been playing with 6 for a while, but also want to get a handle on
Doron Shalvi: that we do plan to use Ocfl, and so just want to see what's going on lately. As we plan our
Doron Shalvi: migration to it.
Rosalyn Metz (she/her): Well welcome.
Rosalyn Metz (she/her): Well, do you want to intro yourself?
Tom: Apologies? I'm still setting up my name's Tom Rebell. I I head up the software development team at Aura, which is the Oxford Research Archive for the University of Oxford. We are Fedora, 3
Tom: for answered or 6 users, and we're currently using or testing Fedora 6 as an Ocfl. And a box implementation for our digital preservation platform.
Rosalyn Metz (she/her): There you come.
Rosalyn Metz (she/her): Neil, will you go next?
Neil Jefferies: We're just introducing it all right, Hi, i'm Jeffrey from Oxford. I'm one of the editors for a zfl
Rosalyn Metz (she/her): next on my screen is Julian.
Julian M. Morley: Hello, I'm. I'm Julie moly i'm kind of it. Operations at Stanford University Libraries.
Julian M. Morley: I, one of the editors, and I apologize my video, not working. And I'm going to blame Zoom for that
Rosalyn Metz (she/her): always blaming somebody.
Dianne Dietrich: Diane, Would you like to introduce yourself?
Rosalyn Metz (she/her): Great
Rosalyn Metz (she/her): welcome
Rosalyn Metz (she/her): Shinwu Kim? Would you like to introduce yourself?
Shinwoo Kim: Hi, I'm Sheno Kim i'm also from Corona University, and same
Shinwoo Kim: information as not yet.
Rosalyn Metz (she/her): Thank you, and welcome
Rosalyn Metz (she/her): so, as Seth mentioned.
Rosalyn Metz (she/her): he is going to do a demo
Rosalyn Metz (she/her): before I move to that. Are there any? I feel like that was also sort of community updates to what people are doing. But are there any others that we may be missed.
Rosalyn Metz (she/her): Okay, let's get to this because I'm really excited, for
Rosalyn Metz (she/her): you can't tell
Rosalyn Metz (she/her): Seth, If you need to share your screen, you should have the ability to. But let me know if you can't.
Seth Erickson: Okay, yeah. Again, I I want to preface all this by saying: this is all very much work in progress. I know I've been. I've been working on this sort of go implementation of Ocfl for a number of years, and I have yet to kind of get to like a a release that I am like really want to call a release. but
Seth Erickson: anyways, I think I need to bite the bullet at a sooner rather than later, and and and I and get something out. So i'm I want to share what I've been working on with Ocfl index, which is really a kind of a layer. On top of this Go, ocf. An implementation that is
Seth Erickson: oriented around access. so
Seth Erickson: let me. There are some slides in the in the in the agenda that you can refer to if you want.
Seth Erickson: I'm going to kind of go through these, and i'll share my screen, I guess.
Seth Erickson: So.
Seth Erickson: and make sure there are any embarrassing tabs up. Everything looks good.
Seth Erickson: Can you see that?
Seth Erickson: All right? So ocfl index has been a project where I've been kind of toying with different ideas around accessing Ocfl repositories
Seth Erickson: in slack. A little while ago I was sort of presenting it as a as really a web-based front end. for Ocfl repositories and I think that actually, that's really not the direction i'm gonna be taking I I I was going down a kind of a web ui rabbit hole that I found really kind of
Seth Erickson: cumbersome and not productive, and I it wasn't the direction. Ultimately, I think i'm interested in going in.
Seth Erickson: So what Ocfl index is right now is really a Api layer on top of an Ocp repository, and again, it's focusing on
Seth Erickson: access. Some it's only read only you don't need any right permissions to your underlying storage layer
Seth Erickson: to use it.
Seth Erickson: so it has 2 goals. One is to enable quick access to information about arbitrary paths in a given objects logical state. So i'll try to unpack this a little bit in an Ocfl object.
Seth Erickson: You have multiple versions, and those versions have logical states which is different than the you know, the the concrete, the content state of the what's stored on
Seth Erickson: the storage layer? and so what I want to make it. I want to make that logical state more accessible, and not just the file pads themselves, but also directory. so, for example, if you wanted to list the top level directories of version, 2 of an object. Or if you wanted to list the files in the folder foo
in a in a directory
Seth Erickson: and an object.
Seth Erickson: that's those are the kinds of access queries that i'm interested in in in supporting. So get state object, I Id version path is kind of the the core
Seth Erickson: function. that I want to implement.
Seth Erickson: And all of this is really to get to that point of making that function possible and and fast, so that it's sort of on the order of a millisecond or so to get that information out of the out of the the repository.
Seth Erickson: so you don't want to have to download the whole inventory, parse it, and then to, and then present that information that would be too slow. So the first step was to build a an indexing database.
Seth Erickson: I've been using sequel light, and in order to make this sort of manageable size.
Seth Erickson: it's storing the state as a hash tree, so that if you have successive versions with the same content
Seth Erickson: you're not adding a lot of new nodes or fields to the database. So each so, as you can see here, we have a version of what re version one and version 2 version 2 has a new read me, but it had, and it has a a directory that has changed names.
Seth Erickson: But the Directory that's stored in the database, the Directory information that's stored in the database remains the same.
Seth Erickson: and to do that i'm calculating digests not just for the files, but for the directories. And and so the directories have a recursive digest that's based on the content of the file, and there's some subtleties there.
Seth Erickson: that we could talk about later. But i'll, for now i'm just gonna kind of gloss over this with some of those those details.
Seth Erickson: so this is a graph structure. The core of the index is just 2 tables, nodes and edges, nodes and names.
Seth Erickson: so a node is represents some unique content, either
Seth Erickson: a file or a Directory. It has a a it is a Boolean about whether it's a directory or not.
Seth Erickson: there's a digest that's stored as robots to keep the size. A little lower
Seth Erickson: and and then there's the names table, which is names are path, elements, file, names, or directory names, and they are have nodes as parents and nodes as children.
Seth Erickson: So I wanted to give some size information. This is not a as done or as complete a slide as I wanted it to be. I think, in my experience the the indexes had this: the size of the index files has been acceptable. so, for example, I've
Seth Erickson: I've run tests where I have a 100,000 objects.
Seth Erickson: and that generates about a gigabyte of of sequel light. data.
Seth Erickson: I think Andrew has also used it, and he he might have you know other other things to share about it.
Seth Erickson: in testing situations, so I think the size is manageable. This is definitely something to keep an eye on. The reason that this Isn't super developed is that I've recently decided. I wanted to add file size information to the to the nodes. So there is some kind of wiggling there's some, you know.
Seth Erickson: Some changes are happening here.
Seth Erickson: so that's the first goal of like how I've kind of
Seth Erickson: It's been working on indexing like building the index itself.
Seth Erickson: but the second part is really around access, and how to support access to different kinds of programming language environments. the vision that I have, and that actually is, you know, I I think I can demonstrate some parts of this vision today
Seth Erickson: is that we have Ocfl index, client libraries. that are implemented ruby, and for implemented in python, or whatever kind of language you want
Seth Erickson: and these will interact with the Ocfl index server which is written in. Go
Seth Erickson: and then the go server will, you know, obviously connect directly to the Ocl. Storage layer.
Seth Erickson: and so I've been kind of learning about Gr. PC. Which is a technology for sorry protocol buffers and Gr. PC. Which are technologies for building
Seth Erickson: services. Api service layers,
Seth Erickson: And
Seth Erickson: so there there are a couple of different options for this sort of this sort of thing right? Well, oftentimes you have.
Seth Erickson: Json Api's, or the conventional approach
Seth Erickson: open api. Is this is a is a standard that will allow you to generate client libraries based on an a specification with just Json. Api
Seth Erickson: I have.
Seth Erickson: I've been exploring Grpc for a couple of reasons that i'll get into. And this is what the service definition looks like in Gr PC. There's a
Seth Erickson: some a set of
Seth Erickson: service definitions like, Get a summary information about the database, about the index list. The objects in the index
Seth Erickson: get information about the version information of an object, and then finally
Seth Erickson: get objects, state
Seth Erickson: and then download the content you can download the content stream that way.
Seth Erickson: So some nice things about. So what I like and what I don't like about the Rpc.
Seth Erickson: It's a really nice way to specify an Api it's simpler than swagger, and open Api
Seth Erickson: has an efficient binary format, and it's really well suited to kind of communication between back end services, which is where I how, which is where, I imagine. Ocf. L index, spinning
Seth Erickson: and most importantly, it allows you to automatically generate client libraries, and and in the the work I've been doing I've been pretty happy with the client libraries
Seth Erickson: and i'll show some of that.
Seth Erickson: What's not so great is that
Seth Erickson: it's not. You know
Seth Erickson: It's not as widely used as traditional Json. The rest Api's and it requires specialized tooling. It can always just use coral.
Seth Erickson: but there are some tools that have emerged. there's a project called buff
Seth Erickson: the Uf. That I have been using that I like a lot.
Seth Erickson: okay. So I think with that i'm just gonna
Seth Erickson: hop into a little demo mode.
Seth Erickson: and we'll see how this goes. I'm. Going to stop sharing for a second.
Seth Erickson: Are there any questions at this point?
Seth Erickson: Okay.
Seth Erickson: So You should see my Vs code
Seth Erickson: window here.
Seth Erickson: and i'm in the project folder for this. i'm gonna just show how you can. There, there's a couple of startup scripts in here that demonstrate how to start
Seth Erickson: the server.
Seth Erickson: it's primarily configured by environment variables. And then you can specify whether or not you want to index file sizes
Seth Erickson: so I can start. I'm going to start the
Seth Erickson: server.
Seth Erickson: and it will complain because there's already something wrong with it.
Seth Erickson: I think we can get that.
Seth Erickson: and there we go.
Seth Erickson: It's great.
Seth Erickson: so I need to. make sure that I have my azure credentials loaded.
Seth Erickson: so it's a little bit early, and I haven't set this up properly, as you can see. Okay, so now what's happened? There is I started the server
Seth Erickson: and it's spit out some some logs. it's connected to my azure bucket.
Seth Erickson: It's created a public data sequelite index.
Seth Erickson: and then it started to index these by these objects, and it found just, you know a handful of them in this demo repository.
Seth Erickson: and so
Seth Erickson: there's all in this project there is a command line client that's called ox
Seth Erickson: and
Seth Erickson: with ox. I can just say aux status.
Seth Erickson: and it will query the server it. The default port is to look at local host 80 80 to connect to the Oc. Of L index, and that's
Seth Erickson: so it's just using the defaults there, and you can see the request appearing in the logs over here. so this is saying, there's an Oc. About repository versus 1.1 with a you know there's a description from the storage route
Seth Erickson: file.
Seth Erickson: there are 10 objects, and it shows when it was last indexed. I can list the objects
Seth Erickson: that shows those 1010 objects.
Seth Erickson: And it shows the versions and the I think the date of the
Seth Erickson: I kind of go back. I think it's the date of the
Seth Erickson: the the the most recent version is what's being listed there?
Seth Erickson: and then I can. Let's say I want to look at a particular
Seth Erickson: objects
Seth Erickson: The command line
Seth Erickson: interface here is not the nicest. It's that hasn't been a focus. It might work so far.
Seth Erickson: but if I wanted to
Seth Erickson: list files.
Seth Erickson: and to go back to those examples of the kinds of queries I was interested in, if I wanted to list the files in the top level of one of these objects.
Seth Erickson: I do Ox Ls: the object Id
Seth Erickson: The default is to look at the head version.
Seth Erickson: the most recent version, and it sees that there is one file in that object
Seth Erickson: which is a tiff file. It has the size, information.
Seth Erickson: and then we have the first couple of characters of the digest
Seth Erickson: If I wanted to save that currently, the only way to do that is with
Seth Erickson: you know. I think I can do cat
Seth Erickson: and then the file name
Seth Erickson: Tab completion would be really nice things to do here.
Seth Erickson: and then I can save that to the file.
Seth Erickson: and I will have outputed my that will
Seth Erickson: might in a while. It's been saved actually, it looks like it wasn't. So there's an escape there.
Seth Erickson: there's no file information actually served.
Seth Erickson: Yeah, you can pop into that.
Seth Erickson: I think the only other thing I wanted to show
Seth Erickson: is
Seth Erickson: the the kind of feature of implementing ruby clients with this.
Seth Erickson: so let me
Seth Erickson: close one of these oops.
Seth Erickson: I'm: going to. Actually.
Seth Erickson: Yeah.
Seth Erickson: I can exit. But it please.
Seth Erickson: I'm gonna stop the server.
Seth Erickson: I'm gonna go into the
Seth Erickson: clients Directory and in the clients directory I have made a ruby gem.
Seth Erickson: that
Seth Erickson: I've installed it. This is a you know, a draft gem.
Seth Erickson: and the code for this is actually incredibly minimal.
Seth Erickson: if you look at the code for it.
Seth Erickson: It is
Seth Erickson: this file.
Seth Erickson: and pretty much this everything here and under this path has been automatically generated, using the protocol buffer, the Grpc part of protocol buffer tools that I've been using.
Seth Erickson: so
Seth Erickson: this example code will connect to an Ocfl repository
Seth Erickson: and which i'm actually running on Fly dev.
Seth Erickson: and it will
Seth Erickson: download the some, and it will access the summary of that repository.
Seth Erickson: So if I run ruby examples summary
Seth Erickson: it should spit out the
Seth Erickson: ruby representation of the object. The the description of of the of the index summary that I showed before, where you have
Seth Erickson: the the version a description of the repository, the number of the objects, and when it was last indexed, and that kind of thing.
Seth Erickson: So this is sort of demonstrating the ease with which you can create client libraries. in different programming languages. So I could also generate a python library.
Seth Erickson: you do have to sort of figure out how those the those Api's work.
Seth Erickson: but so far I've been pleased with this, and that's really what I wanted to show
Seth Erickson: I think i'm stop sharing
Seth Erickson: what i'd like to talk about. A little bit more is
Seth Erickson: just as you can see. There are some rough edges here what i'm looking for is feedback, if you want. If you are interested in in exploring this and trying it out, please do. And
Seth Erickson: you know right issues that kind of thing. if you're interested in using it, I don't think we're there. It's not really ready for that. But I do think it's pretty close. It should be safe to use with your repositories, because you only need to give it. Read access right? It it doesn't require any.
Seth Erickson: in fact, I would recommend not providing credentials to with that. Have right permissions when you're using it right.
Seth Erickson: and then in the that's in the immediate horizon for things i'd like to do to improve it.
Seth Erickson: there
Seth Erickson: and long term, I think the the grpc interface idea isn't is compelling to me, and I think i'd like to expand that to include
Seth Erickson: an Rpc Service definition that includes things like you know, committing new objects like like creating a new version, staging or creating a new version stage modifying that stage and then committing that stage is kind of how I match it.
Seth Erickson: But yeah, that's that's it for me.
Julian M. Morley: And you caught Rosy eating so she can't do the follow up nice. so this yeah, this is really cool.
Julian M. Morley: I have a I I guess. Question about about about flow. Really. I I did something similar to this
Julian M. Morley: except for uglier, and it was for mo app not for Cfl.
Julian M. Morley: And large jokes that we were in in in fashion. Python. but same logical idea, right?
Julian M. Morley: Have you
Julian M. Morley: one thing that one thing that the the way how I did the initial query was, I went to the S. 3 source first, and I just got the inventory files down and
Julian M. Morley: and answer the question from that, and then I and then I stacked that inventory information in the local cash. And then, if I, when I made a subscription query to the object. I checked the cache first. Is it there? If, and is it still current? And if not, then I went to s 3 to get. You know more up to that information.
Julian M. Morley: rather than trying to index everything
Julian M. Morley: all in one go.
Seth Erickson: yeah, that makes sense. I right? Can I, or sorry?
Julian M. Morley: Do you want me to respond to that? Or do you have
Julian M. Morley: of the latency with creating this the object? Still, we are indexing large
Julian M. Morley: collections. Right? So in our case we have, for you know, a couple of You know the paths. It's probably like 7 million paths per per object store, and doing that over https that start up is essentially in hospital. It takes days to index.
Julian M. Morley: So.
Julian M. Morley: I was
Julian M. Morley: wondering if what i'm trying to say is like
Julian M. Morley: this is really cool. I'm concerned that you're going to get to a point where you'll find that you're not able to do that initial indexing step on larger repositories.
Seth Erickson: Yeah.
Seth Erickson: that makes sense. so I it's true. I haven't
Seth Erickson: I think the the biggest
Seth Erickson: Hmm.
Seth Erickson: Well, there's a let me let me get to the different. Let's talk about the different stages of the indexing process. and so there are 3 really 3 stages to it. The first is to scan the Ocfl repository for
Seth Erickson: objects. So one of the things I want the index to provide is is a list of the objects that exist right? So in the scenario that you described
Seth Erickson: it doesn't sound like you have an in a list of everything. To begin with, you kind of have to say. I know this object exists. Please give me information about it.
Seth Erickson: so I I do want to cfl index to at least have
Seth Erickson: the ability to
Seth Erickson: to have a comprehensive list of everything that's there
Seth Erickson: right, even if it's just a list of Ids of object, Ids right, without any much additional information.
Seth Erickson: So to do that
Seth Erickson: you need to scan
Seth Erickson: the repository up to a certain layer where you're finding the the object definition file and say, okay, here's that object definition file. That's an object. Save the path.
Seth Erickson: We don't and so actually, that's a good point. You don't have the Id. You only have the path.
Seth Erickson: So you at least know that there is an object at that path.
Seth Erickson: I think that's the minimal
Seth Erickson: that's the initial indexing phase. It's like we have these objects of these paths. We don't know their ids.
Seth Erickson: The next layer of that is to say, okay, i'm going to download just the inventory file.
Seth Erickson: and then I've got then I've got the the Id the path
Seth Erickson: they'll and all the logical state information that that is going to consume a lot of the storage space of the index.
Seth Erickson: But even that is actually not the slowest part, because I really want file size information. Then the file size information you have to. Then go again. Read the Content Directories
Seth Erickson: you have to do a redirectory of of you don't have to
Seth Erickson: like stat every file. But you do need to do a redirectory on on the the directories themselves, and then you would get the file size information. So that's why i'm
Seth Erickson: that's an option. It's not necessarily a
Seth Erickson: it. It's not the fastest, so I think, but in terms of the the approach that you described as, let's say initially, we have a list of objects, and hopefully, we can do that fast, because I this is something that you know go has fantastic support for concurrency. I do use concurrency here. I could do a better job at a
Seth Erickson: concurrently mapping the storage route. so it's it is doing
Seth Erickson: like. I think I can set the concurrency to, you know 12 simultaneous go routines, or something like that. If I have a let me explain. One of the
Seth Erickson: the default is to look at the number of
Seth Erickson: cores that you have on your processor, and it will use that many go routines to scan the index.
Bye.
Seth Erickson: Okay, I'll I'll
Seth Erickson: Daniel. What? What's your question?
Daniel Salwerowicz: yeah. I was gonna ask as if you had it in presentation. You mentioned it now now to is, what's the reasoning behind to wanting to have the file size information about the files like we have anything specific in mind for that.
Seth Erickson: yeah, I mean, I think the file size information is a good way to like
Seth Erickson: when you like the the what i'm interested in is supporting situations where
Seth Erickson: you know your
Seth Erickson: you've got some data set, and that that data set. And and you're interested in using that data set. And you and you want to maybe explore it. But you not Don't, necessarily want to download the whole thing. It's good, because these files that i'm per i'm
Seth Erickson: imagining working with might range from, You know, a few kilobytes to like, you know, dozens of gigabytes I just knowing what you're getting into when you start to download something would be helpful. And right now there is no way to do that right?
Seth Erickson: without the file size information.
Daniel Salwerowicz: Yeah, yeah. I I was wondering, because in our situation we haven't gotten actually to the point of using all Cfl. Yet. But we it because we needed to very quickly
Daniel Salwerowicz: transfer f
Daniel Salwerowicz: files we're producing from one storage system to the new one. So we got to get a quick and dirty solution, and the first improvement we are trying to do is to at least get
Daniel Salwerowicz: the file says information along with us, but we put it in a regular SQL database. Just so. We know about how much storage space we're using. Since we have quite a very data set, we have
Daniel Salwerowicz: some things that are pictures, something that are digitized, the books or digitized newspapers, which is a mix of everything really in between. And then, yeah, that's why we wanted to have the
Daniel Salwerowicz: as far as this information. So we know how much space we are using
Daniel Salwerowicz: as well as just having a nice way of
Daniel Salwerowicz: having a like
Daniel Salwerowicz: summary and something we can show to the others like how much data we have stored so far, and daily or weekly in just.
Seth Erickson: But this is definitely, I think file sizes might be something again. Kind of the the model that Julian described as like.
Seth Erickson: you know.
Seth Erickson: don't don't get it until you need it right. don't even like that might be the model that needs to happen at every phase Here it's like. So Don't assume
Seth Erickson: Don't proactively catch or go out and and get that information.
Seth Erickson: because in most cases you're probably not going to be using it. Just get the information that people are querying, and then cache that I think that's a good model to to kind of bake into this.
Seth Erickson: you know, projects.
Daniel Salwerowicz: Yeah, yeah, that's that's that's true. Our cases like just the the leadership want to have a monthly summary of about how much data we have stored and right, and the nice thing about this recursive, the recursive structure that I described. Where you have a folder
Seth Erickson: is a Rec. Has a has a digest to which is based on its contents.
Seth Erickson: that also applies to the file sizes. So that means that you can get you. It's possible to know the the size of a version, one or version 2 of an object right
Seth Erickson: quickly. Yeah.
Julian M. Morley: So that's actually a really cool part of your design. I I love the idea of the digesting the directories. that that changes on a large data structure that's going to give you really fast access to what's changed recently, as opposed to time to troll everything. So that's that's what we really need.
Julian M. Morley: what thing have you for again for logs? For my my stick is, I love large repositories. I deal with you all the time. Right? So Have you considered
Julian M. Morley: You flies in some of the native tools, and I I mean, I know about azure, but I was also because it what they will provide you with the report on the bucket.
Julian M. Morley: and so they will tell you like. Here's a list of all that the keys in the bucket. You know it. You set it up as like a chrome job, essentially, and it'll it'll take like a couple of hours them to a report on it, but they'll they'll end up. Don't be the Csv file into the book it into a reporting bucket. That is basically just a list of all your keys.
Julian M. Morley: and I think you can figure it a bit more. But what you could potentially do is if you're using a repository to support that kind of action.
Julian M. Morley: Have them. Have your Cloud provider
Julian M. Morley: do the indexing of the bucket ahead of time, and then you just pull down that Csv file and parcel locally, and that'll be super fast.
Julian M. Morley: But again it's dependent on the provider, and that Mosari doesn't do it. Amazon does not about 0,
Seth Erickson: right? That's interesting. I didn't know about that. So thank you. I mean, I think what that what that is making me think of is that it would be nice for this, for the index to have some kind of interface where you could say like, here's the list of everything
Seth Erickson: you know have at it. and
Seth Erickson: like that it that that seems like that would get you. You know you could meet in the middle, depending on how
Seth Erickson: different services implement that functionality that you're describing.
Julian M. Morley: Yeah, I'm: pretty. I'm: I'm: relatively certain it's almost entirely, either. I'm: just Amazon and maybe Gcp: that does it? Because yeah, they want to differentiate You use our stuff. Don't use those of the guys stuff
Julian M. Morley: and
Julian M. Morley: large buckets. It is kind of useful.
Rosalyn Metz (she/her): Anybody else have questions
Rosalyn Metz (she/her): now that i'm not chewing.
Andrew Woods: of course. i'm. So
Andrew Woods: I mean. Needless to say that it's super awesome and very exciting. thank you for all the work that you've been doing you. You. You've really been, I think. you know, moving the community forward here, which is inspiring, and
Andrew Woods: i'm not always grateful for the the work that you just consistently are producing anyways.
Andrew Woods: So you know, obviously the model that
Andrew Woods: you were
Andrew Woods: you enabling here with the work that you're doing is you effectively having like an Ocfl server
Andrew Woods: that then services can interact with you right right now. We're at the point where to read. Only server. yeah, it.
Andrew Woods: It will evolve into
Andrew Woods: you the possibility of being a readwrite server
Andrew Woods: A. And and then thinking about our systems so like the our upper level applications that will be interacting with Ocfl servers.
Andrew Woods: so Fedora is one example here on the call.
Andrew Woods: You know what
Andrew Woods: you know what we have at Harvard. which is a custom solution, you know, like I'm: I'm thinking about okay, like, how would, how would this work, or what would we need? Or yeah. But yeah, I I feel like it opens up a ton of possibilities.
Andrew Woods: for different types of applications interacting with
Andrew Woods: you like the the goods.
Andrew Woods: And you obviously want one of the issues, and this is not what I was planning on asking about. But you know one of the issues is round permissions, you, and and it it would. It would be great if
Andrew Woods: you know, if if somehow
Andrew Woods: and you know, this is further down the line. But basically
Andrew Woods: giving access to an Ocf repository, but filtered.
Andrew Woods: you know, based on access
Andrew Woods: rules.
Andrew Woods: Seth, you're about to jump in.
Seth Erickson: Well, yeah, the permissions is a huge topic which I obviously i'm not even touching here, and that's on purpose, because I because I think it is the kind of thing that I can safely defer a little bit, because the the way that there are 2 just to kind of. If you're wondering how permissions would
Seth Erickson: might work in this kind of thing. There are 2
Seth Erickson: general approaches. One is to go the kind of traditional Gr PC route, in which case you use something called mutual Tls, which is where your clients are also have certificates that to that are validated by the server.
Seth Erickson: so that would be one approach. The other approach would be to do like a.
Seth Erickson: you know. Oh, off
Seth Erickson: kind of
Seth Erickson: credential that's authentication bear, or whatever it's called, header in each request, which is the more traditional Json
Seth Erickson: like rest approach. But you could do both of you could do either of those. I'm. Also, it would be interesting to have figure out like maybe you have a model where your object ids have prefixes, and then you want to have different kinds of permissions associated with the prefix
Seth Erickson: something that I haven't talked about here is how you might filter objects based on
Seth Erickson: like if you query, what are the objects in the storage?
Seth Erickson: like? What are the kinds of ways that you might want to filter those
Seth Erickson: that's not something I've implemented at all.
Seth Erickson: so yeah, as you say, there's a lot to consider, for, you know, a real implementation of this kind of thing.
Andrew Woods: Yeah, and and and maybe and and I think it's related where i'm going here a little bit is
Andrew Woods: You use cases from the ever upper level applications, right? And and one specific example would be the Api that you already have to find and thinking about our systems, and you like what Api would, what Api would my system want from such as
Andrew Woods: ocf server?
Andrew Woods: and it it seems like it. It would be helpful if if we could start to maybe formalize
Andrew Woods: you the signatures for
Andrew Woods: it, and you know, I think what you've put in place already is very logical.
Andrew Woods: Yeah, like, Serve the the basic operations that you would want. But it would be interesting to hear from folks if there are other
Andrew Woods: yeah services.
Andrew Woods: yeah, or yeah on on the Api that
Andrew Woods: you would
Andrew Woods: find value in or need
here.
Andrew Woods: And I also see Julian's
Andrew Woods: fingers up.
Julian M. Morley: Yeah, I I it's so. It's sort of a caution retail, I guess. the
Julian M. Morley: I think one thing to bear in mind if we go to further. He's got to decide whether or not this is going to be something that's used by clients to make sort of AD hoc queries of repository, for you know, curiosity or low level, or like like like like tasks versus a source of truth for the repository that's used at an institutional level to
Julian M. Morley: guarantee with certainty that stuff is preserved.
Julian M. Morley: Those are 2 different use cases, and
Julian M. Morley: especially with high latency schools like S. 3.
Julian M. Morley: I with lots of objects in
Julian M. Morley: the drift is not a possibility. It's a certainty between what's in the database versus what's actually on the object store.
Julian M. Morley: and you need to decide which is fine. Having a client side querying tool is awesome.
Julian M. Morley: But as I emphatically not what you should using for other clients to call to get information about the repository from
Julian M. Morley: because that information cannot be trusted to be accurate.
Julian M. Morley: and you, don't know who's going to be consuming it, and they may be consuming it, making assumptions about the accuracy that just doesn't hold true.
Julian M. Morley: so that's that's that's.
Julian M. Morley: Be mindful of of of what your target audience is for this specific tool, and be explicit about about
Julian M. Morley: the the the the restrictions there. so I don't want folks to go ahead and like you know they find simply find some clients all the way down, all querying. This. You know, this data store that's
Julian M. Morley: not accurate. No accurate affection was actually in a repository.
Julian M. Morley: So that's that's all I wanted to say.
Andrew Woods: Hmm.
Andrew Woods: I mean, I I I can't help a wonder
Andrew Woods: if
Andrew Woods: you I mean Basically, Julian, are you talking about the the index Ocfl index in this case?
Andrew Woods: just not reflecting there, you always being up to date.
Julian M. Morley: Yeah, essentially. Yeah. So I the the we have this at Stanford we are for our mob repository. We have a system called Preservation Catalog.
Julian M. Morley: That is like in next data store. that's clearing s repositories. multiple s through repositories and a local disk store to that state of because of the observed objects. And
Julian M. Morley: it is real detailed work to make sure that the data in the database always accurately reflects the content and all the data stores.
Julian M. Morley: and it's a whole other thing. and that is, that is, use case one. And then there is the I'm. A client, and I just want to know
Julian M. Morley: casually what's the deal with this object.
Julian M. Morley: and it's okay, if it's not massively upstate. because i'm not making critical decisions about the repository based on there is also that query.
Julian M. Morley: So
Julian M. Morley: when we implemented preservation couple, we have a lot of discussions about how we can make sure that we always have that. What's in the database? Accurate reflects what's all the data source? That's a big deal, because we're presuming this long term, we kind of the forward and sticks there.
Julian M. Morley: and the model that that the tool to set design is great. It's it's awesome but that
Julian M. Morley: that doesn't fly for for for our used case we there's no way we could actually use the I I it's this is like not even a one, but a release. So like
Julian M. Morley: not a problem. But it's not that. But this is a a use case that you do need to like
Julian M. Morley: really getting to real. It's the weeds about to work out that you're doing it right, because the also doing it wrong is your institution is lost data, and you don't know about it because your data so it's not reflecting reality.
Julian M. Morley: So
Julian M. Morley: be fortunate about that? and so the add on to that is that like sequel, light is great.
Julian M. Morley: Have you considered I? If you did want to go into the server side model way. Do you want to have a level certainty of your repository? I would suggest it tomorrow in Postgres and the the Json data types. because that would make wearing, storing and clearing this information
Julian M. Morley: significantly easier. I think
Julian M. Morley: everyone likes to be told to re architecture. Sorry about that. This is not like this. If your used case happens to 10 towards what we're doing.
Julian M. Morley: I would I would recommend it, and then, if you smell it going with something like postgres, and Jason would probably be the the way to go for the next door.
Seth Erickson: I always start with sqlite because I like just a single process. Yeah, and it's zoom that I and I can I? It is designed so that it it, you know, with it is designed with the the idea that you would want you will. I will probably want to implement the the index as a postgres database.
Seth Erickson: But I mean, I yeah, you're right, I mean as the the question of like the distinction between a kind of access service versus a kind of a robust preservation service. I mean point well taken. this is definitely oriented. I mean, what I have in mind is really a very lightweight access service on top of it. Ocf Repository.
Seth Erickson: Right? That is definitely not. The the idea is not that the index would be would replace the repository as a source of truth about about that
Seth Erickson: about it. State: right?
Julian M. Morley: Yeah, yeah, absolutely. And that makes that perfect sense. and just like
Julian M. Morley: just as long as it's clear to all, If you didn't start developing Api's against that other folks start creating against it. That's where you you potentially could really trouble us. The folks doing the query
Julian M. Morley: know upfront that the thing that they are clearing is not necessarily the source of truth for the for the repository.
Andrew Woods: Yeah. And I and I guess the the question that I would bring is what what's the gap that we would need to fill to enable
Andrew Woods: such a model where
Andrew Woods: where we can trust an Ocfl server
Andrew Woods: as the source of truth.
Yeah.
Julian M. Morley: I could tell you what it took Stanford, and it was 18 months of like a massive team of developers working on it.
Julian M. Morley: But we do those things too. But I mean, could you? Could you at least maybe describe one or 2 of the things that you feel like we're big stumbling blocks in that process. Because i'm curious. Honestly, a lot of it was getting the developers on board and synced up with what it means to actually
Julian M. Morley: preserve content, and to have certainty that the content is being correctly preserved.
Julian M. Morley: And once they they they got over the
Julian M. Morley: you know they had to
Julian M. Morley: understand
Julian M. Morley: the the the full scope of the problem space, so they could then influence the correct solution for it. because that if you don't, if you don't know this is a problem. Then the thing that you've implement is not addressing the actual problem.
Julian M. Morley: So a lot of it was just getting folks to to really grapple with the notion of what digital preservation actually looks like in practice with multiple different repositories. And
Julian M. Morley: yeah, the notion of a single source. And what is what is the right, ma? What? What mob is the correct mobile version is for My, how do we make sure that you know we, we don't have drift between different versions of our policy. Same mob.
Julian M. Morley: that was a lot of of of of mental heavy lifting
Julian M. Morley: and after that implementation was relatively relatively straightforward.
Julian M. Morley: But yeah, it's it's it's it's a surprisingly complex issue.
Seth Erickson: So one of the things that I yeah, I mean one of the things that I think that came out of this work again to the kind of recursive digest idea
Seth Erickson: the recursive digest is, that is a really
Seth Erickson: nice tool for kind of the things that you're talking about, because it allows you to say
Seth Erickson: you know
Seth Erickson: I mean
Seth Erickson: with some confidence that, like you know, 2 objects are the same or not.
Seth Erickson: Yeah. right in terms of their content.
Seth Erickson: So so you can.
Seth Erickson: So Ocf Doesn't currently have a mechanism for for saying these 2 objects are identical.
Seth Erickson: but with the reverse of that digest based upon the versions or something like that.
Seth Erickson: you could do that.
Julian M. Morley: yeah, I I guess theoretically, you could compare the checks of the of the of the inventories.
Julian M. Morley: and if the eventually you can't do that well.
Julian M. Morley: and it's not. It's not. It's not great. It would show if it was just a copy. If you copied the Directory Age Directory B.
Julian M. Morley: It's a big. It doesn't show. But obviously the the items in the eventually don't have to be in the same
Julian M. Morley: and the same order. So yes, the checks will be different that it would show if it was a copy of the object, not if it was actually like the same content in it.
Julian M. Morley: yeah, so
Julian M. Morley: I was like it's it's it's a interesting. It's a great problem to to to to to to sink your teeth into
Julian M. Morley: I just it's our experience was it was more work than we thought.
Julian M. Morley: and there's a lot of great learning opportunities there.
Julian M. Morley: 2,
Julian M. Morley: and you do end up implementing essentially
Julian M. Morley: a quite complex system. If you're looking at multiple, try to keep content in sync across multiple different
Julian M. Morley: storage roots
Julian M. Morley: which we are.
Andrew Woods: Yeah. And and i'll say in in our used case, we we don't, deal with that problem. We we have one
Andrew Woods: storage target.
Andrew Woods: And then there's another process that handles the policy based replication.
Julian M. Morley: Yeah, I was just gonna like really slam me for not doing preservation correctly that Andrew. So yeah, all the dodged good job.
Andrew Woods: I'm: I'm: i'm curious. So you. there's been some mic hogging going on for yeah, and and I want to make sure there's space for for others to jump in here
Andrew Woods: comments, reflections.
Julian M. Morley: Oh, my God! And it is Stop! Stop! Stop taking it all the oxygen in the room.
Andrew Woods: Why, you think about it. I I am curious just from others on the line here Daniel Norway I mentioned in the project there of of it sounds like migrating your content to you know, a new storage system, and what what's in involved with that? yeah.
Andrew Woods: Shin Wu and Diane, I'm: I'm interested in just kind of the thinking, you know, within your team, on what's happening at Cornell.
Andrew Woods: we don't need to get into it right now. Maybe it's a future agenda.
Andrew Woods: item on the community call. But
Andrew Woods: yeah.
Andrew Woods: the I think it's
Andrew Woods: beneficial for everyone to kind of get a sense of where people are at.
Daniel Salwerowicz: Yeah, I can just give a quick like summary of the problems we are facing. Cause
Daniel Salwerowicz: we have quite an old to that, I said, or like that is, that goes back a long way back. I don't remember the exact year, but I think the some of the oldest files are like from 2,006 or 7. Ish.
Daniel Salwerowicz: I haven't been there for the most of the time. I I only joined in 2,020, and the problem we had for long, long time was that it was quite AD hoc, and
Daniel Salwerowicz: the voluntary volunteer based digital preservation there wasn't like a proper team or people, or even the one person working with that task.
Daniel Salwerowicz: So what ended up happening that in the end the storage system we had was pretty much just a
Daniel Salwerowicz: in some cases that dumping ground for data and objects.
Daniel Salwerowicz: we
Daniel Salwerowicz: in some case we don't even have checksums for the files we have saved there, so
Daniel Salwerowicz: we just can hope and pray that everything is okay and unchanged. And at least, if if it did change, we have 3 copies. So maybe one of them is correct, and we can try and repair.
Daniel Salwerowicz: But
Daniel Salwerowicz: Now we have the big problem of trying to go to the new
Daniel Salwerowicz: so which system and our goal there is to
Daniel Salwerowicz: first of all take over the
Daniel Salwerowicz: new influx of data we are producing or getting from the others.
Daniel Salwerowicz: and at least the massage it, and
Daniel Salwerowicz: kind of put enough information on to the objects we get.
Daniel Salwerowicz: so we can
Daniel Salwerowicz: later on go back to it.
Daniel Salwerowicz: re-engested properly put it into an all Cfl. Object. Get an inventory for that, and then store it back
Daniel Salwerowicz: with proper metadata. And all of the
Daniel Salwerowicz: the data is running the old Cfl objects.
Daniel Salwerowicz: and then what we are
Daniel Salwerowicz: at least come to this part. We would like to take the old data and kind of try and reprocess it. 150,
Daniel Salwerowicz: and
Daniel Salwerowicz: feel extra information. Make a proper mets. Files, like most of the files, have have a match, but it's out either outdated or
Daniel Salwerowicz: not the quite the information, rich as you would like it to be, so that that it was involved
Daniel Salwerowicz: lots of reprocessing time
Daniel Salwerowicz: and putting it into the new storage system.
Daniel Salwerowicz: And,
Daniel Salwerowicz: we are noticing now that we talk with groups of people that work directly with the physical
Daniel Salwerowicz: Erez agmoni, objects like newspapers, books, films music that we are digitizing and storing a copies of preserving them.
Daniel Salwerowicz: That,
Daniel Salwerowicz: one of the biggest challenge for us is going to be updating metadata because
Daniel Salwerowicz: we see that lots of data we met that we get from production. Isn't either
Daniel Salwerowicz: isn't complete
Daniel Salwerowicz: or isn't fully correct in some cases which will require us to develop a
Daniel Salwerowicz: way of
Daniel Salwerowicz: easily being able to update that. So that's the big
Daniel Salwerowicz: part of the
Daniel Salwerowicz: agenda for the next months, maybe one years to get that working nicely.
Andrew Woods: maybe even years.
Daniel Salwerowicz: Yeah. Worst case, worst case.
Daniel Salwerowicz: But we're also hoping that we're gonna get the help from the teams that directly
Daniel Salwerowicz: produce the digital objects, and
Daniel Salwerowicz: we are trying to be nice with them, and instead of requiring them to make proper full maths, files for us 250.
Daniel Salwerowicz: We are
Daniel Salwerowicz: developing a Jason schema
Daniel Salwerowicz: that they can use to provide us with the information needed
Daniel Salwerowicz: that we can then transform into a proper match file and update it with technical metadata and
Daniel Salwerowicz: all the other metadata we want, but that we at least get proper technical and descriptive metadata that we can get from them.
Daniel Salwerowicz: and we noticed that getting them to
Daniel Salwerowicz: understand maths and give us proper maths was more work
Daniel Salwerowicz: down into the fourth.
Andrew Woods: Yeah. And and I I have a a list of follow-on questions which I will refrain
Andrew Woods: from asking just for the sake of time. Here. but
Andrew Woods: I' I didn't have. Yeah.
Andrew Woods: you're just finalizing the demo
Andrew Woods: that that was awesome. And and thank you. I didn't know if there's any you any any you know, summary or parting, you know, comments that you wanted to make based on the demo.
Seth Erickson: you know. It got pulled in a few different directions here at the end. Oh, thank you for the discussion that definitely gave me a lot of you know, Feedback and I you know I would be honored if any of you would, you know, kick the tires on this thing in any capacity whatsoever?
Seth Erickson: so thank you for
Seth Erickson: for for listening
Seth Erickson: after the feedback.
Rosalyn Metz (she/her): Thanks for coming and doing the demo. Seth. It's appreciated.
Rosalyn Metz (she/her): Well, we have exactly 1 min left.
Rosalyn Metz (she/her): so
Rosalyn Metz (she/her): I don't know if we have time for anything else. But.
Rosalyn Metz (she/her): Julian. You can't just
Rosalyn Metz (she/her): say to that, because the meeting something
Julian M. Morley: I'm using technology. It's what I do.
Rosalyn Metz (she/her): all right. Well, thanks everybody for coming. And thanks again for doing the demo. I'm gonna
Rosalyn Metz (she/her): the app recording.