Harbor Island was created from the rock ballast that was of loaded from California when the ship come up empty..
Actually that part's not true. I recently wrote a book that included some of Seattle's history. Harbor Island and much of what is now the industrial area south of the Seattle downtown was actually underwater most of the time. Elliot Bay came right up and lapped against the bluff that the elevated I-5 runs along today.
In the late 1800s or early 1900s (I'd have to go back to the book to get the exact date) a proposal was made to connect Lake Washington to Puget Sound. This was prior to the Lake Union Ship Canal, locks, and Montlake Cut. The proposal called for digging a canal through what at the time seemed like the shortest route, from Elliot Bay immediately south of Seattle through the narrow valley that I-90 runs through today to join I-5, and through the ridge between the Rainier Valley and the lake. In essence the same route followed by I-90 today.
Work was started on this cut and it went on for some time until it began to be obvious that this route, while shorter, was way too much effort and would not be cost-effective. But quite a bit of the west end of the canal was dug before they halted the project and the spoils all went into the bay and formed a good part of what today is Harbor Island.
Over the following years other dredging and earthmoving projects like the huge Denny Regrade also resulted in dirt that was added to Harbor Island and the South Seattle industrial area. The Denny Regrade project alone resulted in hundreds if not thousands of barge-loads of what used to be Denny Hill that were towed into the south bay and dumped.
So ships calling at Seattle in ballast undoubtedly contributed to the massive landfill in the south bay, but compared to the huge earthmoving projects like the original Lake Washington-Elliot Bay canal and the Denny Regrade, their contribution would not have been much.
First photo below is the Seattle waterfront in 1881. The hill directly behind the waterfront is Denny Hill and it is where the downtown core of Seattle sits today. Next two shots are the Denny Hill Regrade in progress in the early 1900s. Almost the entire hill was removed using the same kind of hydraulic monitors that were used in mining. House owners who refused to sell their property to the city were simply bypassed and their houses isolated on vertical mounds of earth. Eventually, of course, they sold. Some of the nicer homes were moved but most of them were simply blasted to bits by the monitors.
The dirt and debris from the removal of the hill was carried to the waterfront in railcars or in long sluices where it was dumped into barges like the one in the last photo. The barges were hauled to the south end of Elliot Bay and dumped where the spoils from the earlier Lake Washington canal project had been dumped.