Oxide and Friends

Bryan and Adam offer a rebuttal to the AI doomerism that has been gaining volume. And--hoo-boy--this one had some range. Heaven’s Gate, ceteris paribus, WWII, derpy security robots, press-fit DIMM sockets, async Rust, etc. And optimistic as always: the hardware and systems AI doomers imagine are incredibly hard to get right; let’s see AIs help us before we worry about our own obsolescence!On this episode Bryan Cantrill and Adam Leventhal were on a rant; but we welcome others on-stage!Some of the topics we hit on, in the order that we hit them:How we got here: Tweet from Liron ShapiraComet Hale-BoppHeaven's GateCross price supply elasticity of copper and molybdenum marketsCeteris paribus -- Bryan's exit from economicsChris Dixon's book releasing in March 2024 (NOT AN ENDORSEMENT)British to American translation guide"It's not just human-level extinction... it's like potential destruction of all value in the light cone" - Emmett ShearVingian SingularityOxide and Friends: Tales from the bringup labOxide and Friends: More tales from the bringup labBullying self-driving carsAI Resistance Reservists: "For the Lightcone!"Samsung security robotsOxide and Friends: Does a GPT future need software engineers?"I for one welcome our new AI overlords"If we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!

Greg and Rain from the Oxide team joined Bryan and Adam to talk about powerful methods of verifying software: formal methods in the form of TLA+ and property-based testing in the form of the proptest Rust crate. If you care about making software right, don't miss it!In addition to Bryan Cantrill and Adam Leventhal, we were joined by Oxide colleagues Greg Colombo and Rain Paharia.Some of the topics we hit on, in the order that we hit them:Distributed SagasSteno -- Oxide's implementation of distributed sagasLearn TLA+Hillel Wayne talksHillel Wayne on Alloy 6Quickcheck Paper (2000)Proptest docsRain's example codeuse proptest::prelude::*; use proptest::collection::vec; proptest! { #[test] fn proptest_my_sort_pairs(input in vec(any::<u64>(), 0..128)) { let output = my_sort(input); for window in output.windows(2) { assert!(window[0] <= window[1]); } } #[test] fn proptest_my_sort_against_bubble_sort(input in vec(any::<u64>(), 0..128)) { let output = my_sort(input.clone()); let bubble_output = bubble_sort(input); assert_eq!(output, bubble_output); } // These proptests implicitly check that my_sort doesn't crash. }buf-list crateguppy crate... and stay tuned for an upcoming episode revisiting async/await in RustIf we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!

Jordan Hendricks joined Bryan and Adam to talk about her work virtualizing time--particularly challenging when migrating virtual machines from one physical machine to another!We've been hosting a live show weekly on Mondays at 5p for about an hour, and recording them all; here is the recording from June 12th, 2023.In addition to Bryan Cantrill and Adam Leventhal, we were joined by Oxide colleague Jordan Hendricks.The (lightly edited) live chat from the show:DanCrossNYC: The TSC ticks at a fixed rate now days, regardless of voltage scaling on the CPU.jbk: just x86 doesn't provide a consistent want to determine what the rate isjbk: (I guess some chips will tell you via CPUID, but I've yet to actually encounter such chips)jbk: some hypervisors will tell you via an MSRzorg24: Looks the Linux kernel docs have some documentation on the x86 TSC and PIT https://www.kernel.org/doc/html/next/virt/kvm/x86/timekeeping.htmlDanCrossNYC: CPUID or an MSR, but yeah, most systems sample over a fixed interval (determined by another time source) to figure it out.jbk: no, versus some other present component that allows you to measure the frequencyDanCrossNYC: No, the PIT or HPET or something.jbk: https://src.illumos.org/source/xref/illumos-gate/usr/src/uts/i86pc/os/tscc_pit.c?r=236cb9a8jbk: is how it uses the PITjbk: (the HPET code needs to improve it's accuracy, so it's only used when the PIT isn't there at the moment)jbk: some Intel NUCs have no PITjbk: so HPET is the only optionbcantrill: https://github.com/illumos/illumos-gate/commit/717646f7112314de3f464bc0b75f034f009c861eDanCrossNYC: Two big ones: system maintenance without disturbing guest workloads, and also load balancing across a rack."Sevan: ah, thanks.https://github.com/illumos/illumos-gate/blob/717646f7112314de3f464bc0b75f034f009c861e/usr/src/test/bhyve-tests/tests/common/common.c#L166"bcantrill: https://github.com/oxidecomputer/tsc-simulator/tree/masterDanCrossNYC: The guest may well be running NTP itself.iangrunert: I assume you could also check that NTP is alive / has synced recently before doing a migration right?aka_pugs: Do people use IEEE 1588/PTP in datacenters? Maybe finance wackos?zorg24: also it might be tricky to check if NTP synced recently if it is happening in usermodeiangrunert: Might've missed this - is it just the hypervisor that has to run NTP recently or the VM as well?saone: I believe it was just the hypervisorDanCrossNYC: The host.DanCrossNYC: A guest may or may not; that's up to the guest.jbk: but IIUC, if the guest IS running NTP, then the host definitely needs it to avoid any time warpsDanCrossNYC: Yup.DanCrossNYC: Fortunately, there's a bit of an out for the blackout window during migration: SMM mode can effectively pause a machine for an indefinite period of time.DanCrossNYC: We don't USE SMM anywhere, but robust systems software kinda needs to handle the case where the machine goes out to lunch for a minute.zorg24: 🙌 hooray for hardware with no SMM useDanCrossNYC: We have done everything we can to turn it off.ahl: https://github.com/dtolnay/case-studies/blob/master/autoref-specialization/README.mdahl: https://github.com/oxidecomputer/propolisearltea: it worked so well I almost thought the VM didn't migrate 😅saone: It's easy to forget that there's a world outside the cloud, but edge deployments that have physical peripherals hooked up need to maintain those connections to peripherals; migrating those peripherals to cloud environments and managing that integration has been a big challenge for my group.iangrunert: https://signalsandthreads.com/clock-synchronization/ Good listen about clock synchronization and PTP in the ""finance weirdos"" world. MiFID 2 time sync requirements require timestamping key trading event records to within 100 microseconds of UTC.jhendricks: a bit belated, but the propolis side of these changes: https://github.com/oxidecomputer/propolis/commit/7ed480843d3b5cfd9fd07dce41772f8eac4e9171saethlin: The calvalry??saethlin: Are we just going to let that slidesaethlin: Is this a pronunciation situation againzorg24: not the first time I've heard it pronounced that way 🤷saethlin: Well maybe it's me learning this timeDanCrossNYC: CalvaryDanCrossNYC: That's the religious thing.ahl: https://github.com/illumos/illumos-gate/blob/0c5967db436935325af441af2b27d337f4e64af5/usr/src/uts/common/os/cyclic.c#L44zooooooooo: thought this was rust typescript at first 😳DanCrossNYC: Dunno... I missed it. 🙂ahl: * Starting in about 1994, chip architectures began specifying high resolution * timestamp registers. As of this writing (1999), all major chip families * (UltraSPARC, PentiumPro, MIPS, PowerPC, Alpha) have high resolution * timestamp registers, and two (UltraSPARC and MIPS) have added the capacity * to interrupt based on timestamp values. These timestamp-compare registers * present a time-based interrupt source which can be reprogrammed arbitrarily * often without introducing error. Given the low cost of implementing such a * timestamp-compare register (and the tangible benefit of eliminating * discrete timer parts), it is reasonable to expect that future chip * architectures will adopt this feature. aka_pugs: Bryan's TSC is overflowing.DanCrossNYC: That's Tom.DanCrossNYC: Riding in with the cavalry.aka_pugs: Good session.ahl: Thanks...

Bryan and Adam are joined by Ashley Williams to talk about open source governance... and the recently, and various stumblings of the Rust project leadership.

Bryan and Adam are joined by Jonathan and Jignesh from Samtec to discuss working together to build the Oxide Rack. We've all seen bad vendors--what does it mean to be a great partner? Also: silicon photonics are (still!) just 18 months away!

Bryan and Adam are joined by Oxide colleagues Arjen, Matt, John, and Nathaneal to talk about the management network--the brainstem of the Oxide Rack. Just as it ties together so many components, this episode ties together many many (many!) topics we've discussed in other episodes.We've been hosting a live show weekly on Mondays at 5p for about an hour, and recording them all; here is the recording from May 8th 2023.In addition to Bryan Cantrill and Adam Leventhal, we were joined by Oxide colleagues Arjen Roodselaar, Matt Keeter, John Gallagher, and Nathanael Huffman.This built on work described in many previous episodes:Cabling the Backplane Prior to going all-in on a cabled backplane with blind-mated server sleds (i.e. no plugging, unplugging, mis-plugging network cables). We (Bryan) espoused an "NC-SI or bust" mantra... at least in part to avoid doubling the cable count. With the cabled backplane, the reasons for NC-SI disappeared (which let the many reasons against truly shine).The Pragmatism of Hubris in which we talk about our embedded operating system, Hubris (and it's companion debugger, Humility). Hubris runs on the service processors that are the main endpoints on the management network. Matt's work controlling the management network switch (the VSC7448) is in the context of Hubris, as is John's work communicating with the sleds over the management network.The Power of Proto Boards showed and told about the many small boards we've used in development. Several of those were purpose built for controlling and simulating parts of the management network.The Oxide Supply Chain Kate Hicks joined us to talk about the challenges of navigating the supply chain. Mentioned here in the context of "supply-chain-driven design": we designed around the parts we could procure! Tip: stay away from "automotive-quality" parts when the auto industry is soaking them all up.Holistic Boot in which we talked about how (uniquely!) Oxide boots from nothing to its operating system and services. Over the management network, we can drive server recovery by piping in a RAMdisk over the network and then (slowly) through the UART to the CPU.Get You a State Machine for Great Good Andrew joined us to talk about his work on a state-machine driven text-UI and its companion replay debugger. We mentioned this in the context of John replaying the long upload process in seconds rather than hours to fix a UI bug.Major components of the management networkMatt's VSC7448 dev kitMatt's remote tuning setup via webcamManagement network debuggingManagement network debugging

Erin Kissane joins Bryan and Adam to talk the new social network "Bluesky" through the lens of her blog post "Blue Skies Over Mastodon". Long-time friends of Oxide and social-media aficionados Time Bray and Steve Klabnik also helped shed light on technical and social aspects of the net network.Blue Skies Over Mastodon (with Erin Kissane and Tim Bray)We've been hosting a live show weekly on Mondays at 5p for about an hour, and recording them all; here is the recording from May 1st, 2023.In addition to Bryan Cantrill and Adam Leventhal, we were joined by special guest Erin Kissane and long-time acquaintances of the show Tim Bray and Steve Klabnik.Some of the topics we hit on, in the order that we hit them:Erin's blog post Blue Skies Over MastodonMastodon blog (5/1) A new onboarding experience on Mastodon]Tim's blog post from November Bye Twitter"Buy the rumor, sell the news"Hellthread"Skeet" is to "Tweet" is to "Toot" (aka "Publish")skyline.gayBluesky blog Composable ModerationLobstersPhanpySo you've been publically shamed by Jon RonsonIf we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!

The Rust Foundation caused a fracas with their proposed new trademark rules. Bryan and Adam were lucky enough to be joined by Ashley Williams, Adam Jacob, and Steve Klabnik for an insightful discussion of open source governance and communities--in particular as applied to Rust.Rust Trademark: Argle-bargle or Foofaraw?We've been hosting a live show weekly on Mondays at 5p for about an hour, and recording them all; here is the recording from April 17th, 2023.In addition to Bryan Cantrill and Adam Leventhal, we were joined by Ashley Williams, Adam Jacob, and Steve Klabnik.Some of the topics we hit on, in the order that we hit them:SuccessionThe Simpsons (explaining the title of this episode)The WireThe Wire at 20 PodcastThe Register: Rust Foundation Apologizes for Trademark PolicyJomboy (our aspiration)Ice WeaselPamela ChestekBryan's talk from Node Summit 2017: Platform as a Reflection of ValuesLinux Foundation form 990Rust Foundation BoardRust Foundation participation rulesIf we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!

Bryan and Adam are joined by Doug Wibben and Robert Keith to talk about the mechanical design of the cabled backplane of the Oxide rack that allows for "blind-mated" server sleds--no network and power cables to plug, unplug, and mis-plug! Watch the chapter art for relevant pictures.We've been hosting a live show weekly on Mondays at 5p for about an hour, and recording them all; here is the recording from April 3rd, 2023.In addition to Bryan Cantrill and Adam Leventhal, we were joined by Oxide colleague, Robert Keith, and special guest, Doug Wibben.00:00 03:02 09:52 11:09 12:16 12:58 ...

Andrew Stone of Oxide Engineering joined Bryan, Adam, and the Oxide Friends to talk about his purpose-built, replay debugger for the Oxide setup textual UI. Andrew borrowed a technique from his extensive work with distributed systems to built a UI that was well-structured... and highly amenable to debuggability. He built a custom debugger "in a weekend"!Some of the topics we hit on, in the order that we hit them:tui-rsCrosstermThe reedline crateEpisode about the "Sidecar" switchElm time-travel debuggingReplay.ioDevtools.fm episode on Replay.ioAADEBUG conferenceCalifornia horse meat lawThe (lightly) edited live chat from the show:MattCampbell: I'm gathering that this is more like the fancy pseudo-GUI style of TUI, which is possibly bad for accessibilityahl: we are also building with accessibility in mind, stripping away some of the non-textual elements optionallyMattCampbell: oh, coolahl: Episode about the "Sidecar" switch: https://github.com/oxidecomputer/oxide-and-friends/blob/master/2021_11_29.mdMattCampbell: ooh! That kind of recording is definitely better for accessibility than a video.uwaces: Were you inspired by Elm? (The programming language for web browsers?)bcantrill: Here's Andrew's PR for this, FWIW: oxidecomputer/omicron#2682uwaces: Elm has a very similar model. They have even had a debugger that let you run events in reverse: https://elm-lang.org/news/time-travel-made-easybch: I’m joining late - 1) does this state-machine replay model have a name 2) expand on (describe ) the I/o logic separation distinction?ahl: http://dtrace.org/blogs/ahl/2015/06/22/first-rust-program-pain/zk: RE: logic separation in consensus protocols: the benefit of seperating out the state machine into a side-effect free function allows you to write a formally verified implementation in a pure FP lang or theorem prover, and then extract a reference program from the proof.we're going to the zoo: lol i’m a web dev && we do UI tests via StorybookJS + snapshots of each story + snapshots of the end state of an interactionig: At that point you could turn the recording into an “expect test”. https://blog.janestreet.com/the-joy-of-expect-tests/we're going to the zoo: TOFU but for tests 🥰uwaces: Are you at all worried that you are replicating the horror that is the IBM 3270 terminal? — I have personal history programming on z/OS where the only interface is a graphical EBCDIC 3027 interface — the horror is that people write programs to interact with graphical window (assuming a certain size).ahl: https://docs.rs/serde/latest/serde/#data-formatsahl: SHOW NOTES Bryan as "semi-elderly" engineerMattCampbell: didn't Bryan write a blog post on this?MattCampbell: http://dtrace.org/blogs/bmc/2008/11/16/on-modalities-and-misadventures/uwaces: https://www.replay.ioahl: https://devtools.fm/episode/9ahl: e.g. https://altsysrq.github.io/proptest-book/intro.htmlwe're going to the zoo: https://github.com/AFLplusplus/LibAFLig: Are you using proptest, quickcheck, or something else?nickik: This really started with Haskell https://hackage.haskell.org/package/QuickCheck Its also cool that it does 'narrowing' meaning it will try to find an error, and then try to generate a simpler error case.endigma: how different is something like this from what go calls "fuzzing"Riking: Fuzzing does also have a minimization stepwe're going to the zoo: https://github.com/dubzzz/fast-checkRiking: Property-based testing tends to be structured differently in philosophy, while fuzzers are more aligned to "give you a bag of bytes"nickik: http://www.quviq.com/products/erlang-quickcheck/endigma: yeah I can tell its a different structure, but the overall goal seems similarwe're going to the zoo: they are nonexclusive approaches to testingpapertigers: I think Kelly was doing a bunch of tests at Joyent based on quick check and prop test. First time I encountered itwe're going to the zoo: libafl provides a #[derive(Arbitrary)] macro that will provide the correct values for a structuwaces: Lots of stuff in Rust existed first in Haskell (build.rs, quote!, Derive macros, Traits, ect….)…nixinator: https://tenor.com/view/%C3%B3culos-escuro-exterminador-terminator-arnold-schwarzenegger-gif-14440790we're going to the zoo: “what do these means” depends on who you ask lolwe're going to the zoo: fast-check is 🔥 for TypeScriptendigma: if the tested function is deterministic and the test is testing arbitrary input and testing against the result to be derivative in some way of the input function by some f(x), don't you end up re-implementing the tested function to provide the expected result? how does the author choose what properties of a system to test without falling into a "testing the test" pit?we're going to the zoo: Rust: “Here comes the Haskell plane!”nixinator: Isn’t rust == oxidationendigma: yesendigma: in a scientific sensenixinator: Iron oxide 🙂 lolnixinator: Very good!GeneralShaw: Is prop test a way of formal verification? Is it same/different?ahl: https://dl.acm.org/conference/aadebugig: I mean, Haskell is an academic rese...