Previously, the oldest crust samples we have on record date back to 2.7 billion years but pieces of earlier crust have remained elusive.
The geologists discovered the chunk of rock enveloped by granite in an area known as "The Canadian Shield" which forms the ancient geological core of the North American continent.
The roughly 4.3 billion-year-old volcanic rock is basalt and Mr O'Neil believes was once underneath the Earth's first oceans.
The researchers were able to date the rock using a new technique that has only existed for about the last decade. By measuring an isotope that was only produced within the Earth's first 500 million years, called neodymium-142, researchers were able to determine that the sliver of Earth had formed between 4.2 billion and 4.3 billion years ago.
The finding sheds some light on the early development of our planet and gives geologists helpful insight into the planet's early geodynamics.
The discovery could even be useful in helping us to better understand other planets in our solar system, Mr O'Neil said.
"If we understand early processes that shape our planet, we can maybe understand other planets: Why are they different? Or are they similar and where in their life they drift apart in terms of geology?"