If you kept heading east in the USA, and followed the coast to its most eastern point, or even hopped on a boat to go to one of its territories such as Guam you still would not have made it to its most eastern point. To reach that, the quickest direction to head in is West.
It is, of course, all to do with the 180th meridian, the arbitrary line used as the basis of the international date line. At the International Meridian Conference in Washington in 1884, astronomers and representatives from 25 countries chose the meridian of the Royal Observatory at Greenwich as the "prime meridian", the point at which the longitude is set at 0°. Prior to the convention, called by US President Chester Arthur, there wasn't much in the way of agreement as to where this prime meridian should be set. France published maps with the prime meridian going through Paris, while China created maps with 0° going through Beijing, something of a nightmare for anyone wishing to navigate their way across the world.
This line, which separates what we call east and west, places the volcanic Alaskan island of Semisopochnoi less than 16 kilometers (10 miles) west of the 180° meridian, making it – to the west of the most western part of Alaska – the most eastern part of the USA.
If for some reason you don't count the volcanically active island, or international territories, the next most-eastern point is Sail Rock off West Quoddy Head in Maine, on the other side of the USA.
[H/T: Mental Floss]