Rust 1.91: Num-bigint-dig Warnings In LanceDB

Hey everyone! So, we've stumbled upon a bit of a snag when trying to build the latest LanceDB (version 0.22.2) using the shiny new Rust 1.91.0. It seems like a dependency, specifically num-bigint-dig version 0.8.4, is throwing some future-incompatibility warnings. Now, I know what you're thinking: "Warnings? Big deal!". But guys, these aren't just any old warnings; these are the kind that signal that something in the code is going to break in a future Rust release. It's like getting a heads-up that a bridge you're using is scheduled for demolition – you can still cross it for now, but you don't want to be on it when they bring out the wrecking ball!

This whole situation is a classic example of how deeply intertwined dependencies can get in the Rust ecosystem. The num-bigint-dig crate isn't directly used by LanceDB. Oh no, that would be too simple! Instead, it's being pulled in transitively. That means LanceDB uses rsa, which then uses reqsign, which finally pulls in opendal. And somewhere down that chain, boom, there's our culprit, num-bigint-dig. It's like a game of dependency dominoes, and we've hit the first one that's about to fall.

Right now, building LanceDB 0.22.2 on Rust 1.91.0 still works. Your builds will pass, and everything will seem peachy. But this is a ticking time bomb, my friends. When Rust eventually decides to make these warnings hard errors (which they will!), any project using this specific dependency chain will suddenly find themselves unable to build. This could mean major headaches for anyone relying on LanceDB for their data projects. We're talking about potentially broken deployments, stalled development, and a whole lot of frantic debugging to figure out why everything just broke.

The Nitty-Gritty: What's Happening Under the Hood?

So, what exactly is num-bigint-dig doing that's causing all this fuss? The cargo report future-incompatibilities command, which is super handy for sniffing out these kinds of issues, points the finger squarely at num-bigint-dig v0.8.4. The specific warning we're seeing is: warning: macro vec is private. This might sound a bit technical, but essentially, the num-bigint-dig crate is using a macro called vec in a way that Rust is planning to disallow in the future. Previously, Rust was a bit more lenient, allowing macros marked as private to be used in certain contexts. However, they're tightening things up to enforce better encapsulation and prevent potential ambiguities. The num-bigint-dig crate is using this vec macro in several places, including within its biguint.rs, prime.rs, and bigrand.rs files.

Each of these instances is flagged with a note pointing to issue #120192 on the Rust GitHub repository. This issue details the change and why it's happening – basically, to ensure that macros behave as expected and don't leak implementation details. For us users, it means that while our code compiles today with Rust 1.91.0, it's relying on behavior that's going away. Think of it like using an old-fashioned phone booth – it works, but nobody's building new ones, and eventually, they'll all be gone.

This transitive dependency chain is a common beast in software development. We've got lancedb at version 0.22.2, which relies on lance (0.38.2). lance itself pulls in lance-io (0.38.2), and this is where opendal (0.54.1) comes into play. opendal then depends on reqsign (0.16.5), and that's the one bringing in rsa (0.9.8), which finally brings num-bigint-dig (0.8.4) into the party. It's a long, winding road, and the problematic crate is pretty far down the line, making it harder to spot immediately.

Our Environment and the Players Involved

Let's break down the cast of characters in this Rust drama. We're building this on Fedora 42 with the latest Rust 1.91.0 (stable). The key players are:

• lancedb: 0.22.2
• lance: 0.38.2
• opendal: 0.54.1
• reqsign: 0.16.5
• rsa: 0.9.8
• num-bigint-dig: 0.8.4

This is the lineup that's currently triggering the warnings. It's important to note these versions because the issue might be specific to this particular combination. If you're using older versions of Rust or different versions of these crates, you might not be experiencing this specific problem right now. However, as Rust evolves, these kinds of issues tend to surface more broadly, so it's good to be aware of them.

We ran cargo tree -i num-bigint-dig to get a clear picture of how num-bigint-dig is being included. The output, as you can see in the provided details, traces a clear path from opendal back through reqsign and rsa. This is crucial information for anyone trying to debug or fix this. It tells us exactly which crates need to be updated or patched to resolve the issue.

Reproduction Steps: How to See the Problem Yourself

Want to see this for yourselves, guys? It's pretty straightforward. If you have the right environment set up, you can trigger these warnings by building LanceDB 0.22.2 on Rust 1.91.0. The command to generate a report on these future incompatibilities is:

cargo report future-incompatibilities --id 1

This command will scan your project's dependencies and specifically highlight any crates that are using features or patterns that are slated to become hard errors in future Rust versions. In this case, it will confirm the warnings related to the private vec macro within num-bigint-dig.

It's a good practice to run this command regularly, especially after updating your Rust toolchain or key dependencies. It acts as an early warning system, allowing you to address potential build failures before they become critical problems. Think of it as your project's regular health check-up!

The Impact: Why Should We Care?

Now, let's talk about the real impact here. As I mentioned, currently, we're only seeing warnings. Your builds on Rust 1.91.0 will succeed, and you can continue developing and deploying your applications that use LanceDB. However, this is a temporary state of affairs. The Rust compiler is designed to evolve, and warnings like these are signals of upcoming breaking changes. In a future Rust release, these warnings will graduate to become hard errors. This means that any build using this dependency chain will simply fail. It won't compile. Period.

This transition from warning to error is a deliberate part of Rust's development process. It gives developers time to adapt to changes and ensures the stability and soundness of the language over time. But for us on the user end, it means that the current setup is not sustainable. We must address this issue to ensure our projects remain buildable with future Rust versions.

Imagine you have a critical application deployed that relies on LanceDB. You perform a routine update of your Rust toolchain to the latest stable version, expecting everything to work smoothly. Instead, your entire build pipeline grinds to a halt because of a dependency issue that wasn't fixed. This could lead to significant downtime, urgent hotfixes, and a lot of stressed-out developers. Proactively fixing these kinds of issues, even when they are just warnings, is crucial for maintaining the robustness and reliability of your software supply chain.

Furthermore, this situation highlights the importance of keeping dependencies up-to-date. While num-bigint-dig 0.8.4 is the version causing trouble, the real issue might be that an intermediate dependency hasn't been updated to use a newer, compatible version of num-bigint-dig or to work around the issue itself. This is why maintaining a healthy dependency graph is so important.

What We've Tried So Far

We're not just sitting around twiddling our thumbs, folks! We've done some digging to see if we can nudge this issue in the right direction. Our first instinct was to try and update the dependencies that seemed most relevant. We ran cargo update -p opendal. This command tells Cargo to try and fetch the latest compatible version of the opendal crate. However, even after updating opendal to version 0.54.1, we found that it still resolves to the problematic rsa version 0.9.8, which in turn still pulls in num-bigint-dig 0.8.4. So, that path didn't immediately solve the problem.

Next, we tried updating the rsa and reqsign crates directly. The idea here was to see if newer versions of these specific crates were available that either didn't depend on the older num-bigint-dig or had already addressed the compatibility issue. Unfortunately, under the current dependency constraints imposed by opendal (which seems to be fixed on reqsign 0.16.5), we couldn't find newer, compatible versions of rsa or reqsign that would break the chain leading to the problematic num-bigint-dig.

It's worth noting that reqsign itself does have a newer release (0.18.0). However, the opendal version we're using (0.54.x) has a constraint that locks it to reqsign 0.16.x. This is a common scenario in dependency management: a top-level crate might be compatible with newer versions of its dependencies, but a crate further down the tree is holding things back by depending on an older version of an intermediate crate.

This is where things get tricky. We're essentially stuck because of version pinning or compatibility requirements further up the chain. It highlights the delicate balance required to keep dependencies updated without introducing regressions.

The Path Forward: Our Request

So, what's the game plan? Our primary request is for the maintainers of the LanceDB project (or the relevant upstream crates like lance-io or opendal's integration) to please consider bumping the dependency chain. This means updating opendal to a version that is compatible with newer versions of reqsign, which in turn can then use a newer rsa, and ultimately, a version of num-bigint-dig that has resolved the future incompatibility warnings. We need to break this chain that's leading us to the problematic num-bigint-dig 0.8.4.

Ideally, this involves checking for newer releases of opendal and its dependencies, and if necessary, updating them to versions that are compatible with current Rust standards and don't pull in outdated or problematic crates. When upstream dependencies are updated, it resolves issues like this for everyone downstream.

We understand that maintaining these complex dependency graphs can be challenging. If you'd prefer, we are absolutely willing to file upstream reports ourselves for opendal, reqsign, or rsa, or even num-bigint-dig if that's deemed the most appropriate route. Just let us know which path you think is best, and we'll happily contribute to getting this resolved. Our goal is to ensure that LanceDB remains robust and future-proof for all its users.

Alternatively, if an upstream fix isn't immediately available or feasible, we could explore using Cargo's [patch] section in Cargo.toml to override the problematic dependency. This would involve pointing Cargo to a specific version of num-bigint-dig that has been fixed, or even a forked version if necessary. However, this is generally a last resort, as it can make dependency management more complex. The preferred solution is always to get the upstream dependencies updated.

Let's work together to keep the Rust ecosystem healthy and our builds running smoothly! Thanks for your attention, guys!