High Precision Machining: What Tolerances and Surface Finishes Are Realistic, and Which Ones Can You Verify?

Every precision shop publishes the same numbers: ±0.003 mm on milling and turning, ±0.002 mm on grinding, Ra 0.8 µm and finer where a seal or bearing rides. The number on a drawing, however, and the number you can prove when the crate arrives are two different things. This guide is written for the buyer who wants to know which tolerance and finish claims on high precision machining parts are realistic in the first place, and, just as important, which of those claims your own gauges can confirm, which ones only a coordinate measuring machine can settle, and what the inspection paperwork must state before any of it counts as verified.

What makes a tolerance or finish claim "realistic" for a buyer?

A claim has two halves, and conflating them is where most sourcing mistakes start. The first half is manufacturing realism: can a stable process actually hold the band run after run, on that feature, in that material? The second half is verification realism: once the parts exist, is there an instrument that can measure the claim to a meaningful fraction of the tolerance, and will the shop put that reading on a report? A tolerance that is machinable but not verifiable is a claim you can never collect on. A finish that is real but specified so loosely that no two instruments agree on it is a specification nobody can enforce.

For most engineered parts the realistic territory is well understood. Milled and turned features on a disciplined CNC floor hold ±0.003 mm; precision-ground bearing seats, journals, and seal faces hold ±0.002 mm. Surface finish follows the same ladder — Ra 3.2 µm from standard milling or turning, Ra 1.6 µm from fine work, Ra 0.8 µm from precision grinding, and finer values from grinding plus lapping or honing. Those bands are the honest envelope, and this facility states them plainly on its machined parts page alongside the process that holds each one.

The buyer-side question this guide pushes on is narrower: of everything printed on the drawing, what can you independently confirm at your own receiving dock, what requires the supplier's CMM report, and how do you read that report so a pretty number does not pass a claim that was never really measured? The sections below walk through the tolerance bands, the form controls, and the surface finish callouts, and for each one, what proof actually counts.

Which tolerances can your own gauges confirm when the parts arrive?

Start with a blunt inventory of what ordinary measuring tools are good for, because the boundary is not where most buyers assume. A digital caliper reads to 0.01 mm and a micrometer to 0.001 mm, but resolution is not the same as trust: the micrometer's real-world uncertainty, once operator technique, temperature, and part deflection are added, lands a good deal wider than its last digit.

Tolerance on the drawing	What a hand tool can honestly confirm	What settles the argument
±0.1 mm and looser (general block)	Digital caliper — comfortably inside its capability	Caliper or micrometer reading is enough
±0.05 mm	Micrometer, with care; caliper at the edge of its resolution	Micrometer at a stable temperature
±0.01 mm	Micrometer only as a quick check, never as final proof	CMM or calibrated bore gauge
±0.003 mm (milled / turned)	Not reliably — hand-tool uncertainty eats the band	CMM report, measured against the CAD model
±0.002 mm (ground)	No — a two-point hand reading cannot carry this band	CMM under thermal control, per-feature

The rule hidden in the table is that the checking instrument should be roughly an order of magnitude finer than the band it verifies. Against a ±0.003 mm band — ten microns wide, end to end — a tool that itself carries a few microns of uncertainty is measuring noise. That is why the facility behind this guide verifies its critical dimensions on Hexagon CMMs rather than at the bench, and why the dimensional report that ships with the batch is the deliverable, not the pass stamp.

Why does a ±0.003 mm band need a gauge that can see a fraction of it?

The arithmetic is unforgiving. A ±0.003 mm callout describes a window only 0.010 mm wide. A digital caliper's last digit is 0.01 mm — one full window per step — which makes it useless for judging where inside that window a part sits. A micrometer reads to 0.001 mm, so it can see ten steps across the window, which sounds adequate until you add the real error sources: how hard the spindle is turned, whether the part and the gauge are at the same temperature, and whether the part surface is clean and round where the anvils touch. Each of those contributes microns, and once they stack up, two careful inspectors can disagree by more than the tolerance they are checking.

This is the measurement-system problem, and it has a standard name: gauge repeatability and reproducibility, or GR&R. A GR&R study takes the same feature, the same gauge, several operators, and repeated readings, and asks a blunt question — how much of the tolerance band does the measurement itself consume? If the measurement consumes a large slice of the band, then parts that "pass" and parts that "fail" are partly a coin flip, and the Cpk number the supplier quotes is fiction built on a ruler that cannot see straight.

The practical consequence for a buyer is not that you need to run GR&R studies yourself. It is that you should treat a ±0.003 mm or tighter claim as a coordinate-metrology claim, not a bench-gauging claim. When a shop says it holds ±0.003 mm on milling and turning, the supporting question is not "do you have good calipers" but "which CMM measures the critical dimensions, at what reference temperature, and can I see a report from a completed job." Answers that name the machine, the temperature, and the document are describing a real measurement chain. Answers that wave at the tool cabinet are describing a hope.

Which form tolerances defeat handheld gauges entirely?

Size is only half of what a drawing controls. Roundness, cylindricity, flatness, straightness, and true position are form and location controls, and they quietly defeat every handheld tool in the box. A caliper measures two opposite points of a diameter; it says nothing about the shape between those points. A bore can be out of round by more than its size tolerance allows — three-lobed, for example, in a classic error that two-point gauging misses entirely — and still measure "on size" at every orientation you check it.

The same logic applies up the chain. Flatness is not a single reading but a relationship between a surface and a reference plane, which is why it is checked with a height gauge on a surface plate or probed on a CMM. True position is meaningless without a datum scheme, because "positioned relative to what" has to be answered before "how far off is it" can be. Roundness of a journal that runs in a bearing matters precisely because the bearing sees the whole circumference, not two opposite points. None of these can be argued with a micrometer, and a supplier that tries to prove them with one is telling you how it actually measures, which is to say, not.

This is where the inspection instrument list on the factory floor stops being a marketing detail and becomes a contractual fact. A facility that verifies dimension, form, and finish with coordinate measuring machines, a profile and roughness machine, and dedicated form instruments is equipped to answer the callouts a real drawing contains. One whose "inspection" is a bench micrometer can answer diameters and little else, so the drawing should be read against the measuring equipment, not against the brochure. The instrument list this floor actually verifies with is documented on the quality control page.

Why can the same surface measure differently on two roughness instruments?

Surface finish looks like the simplest number on the drawing and is often the least reproducible. Ra — arithmetical mean roughness — is a genuine, standardized calculation, but the value it returns depends on how and where you sample the surface. The two common instrument families approach the same texture differently. A stylus profilometer drags a diamond tip across the surface and traces the profile physically. An optical system reads light scattered or reflected from the surface instead. The two see the same texture through different filters, and their Ra values on a real machined surface routinely differ — sometimes by more than a finishing step — without either instrument being "wrong."

The sampling strategy matters just as much as the instrument family. Ra is an average over a sampled length, and the cutoff — the filter that separates the roughness you care about from the waviness and form you do not — changes what the average includes. Measure the same face with different cutoffs or different evaluation lengths and the reported Ra shifts, even on a single instrument. Add the practical question of where on the part the measurement was taken — the middle of a face or its edge, where machining marks and burrs live — and two honest readings of one part can disagree enough to fail one inspector and pass another.

None of this makes Ra a bad specification. It makes Ra an incomplete one until the measurement method is pinned down. A finish callout that says only "Ra 0.8 µm" leaves the instrument family, the cutoff, and the evaluation length undefined, and therefore leaves the door open for a supplier to measure its way to a number that suits it. A finish callout that states the value, the method, and the cutoff is a specification that two laboratories can both enforce.

What should a surface finish callout specify so the number can be reproduced?

A verifiable finish specification carries four pieces of information, and most drawings carry only the first. The value alone — Ra 0.8 µm, say — names the target but not the ruler. Add the instrument family (stylus or optical), the cutoff or sampling length, and the location on the feature where the reading is taken, and you have turned a wish into a test that anyone with the same equipment list can repeat.

• The value and unit. Ra 0.8 µm is the grinding band for bearing seats, seal faces, and running journals; Ra 1.6 µm suits locating faces and good sliding fits; Ra 3.2 µm covers general milled and turned surfaces. Pick the coarsest value the function tolerates.
• The instrument family. State stylus or optical. The choice changes the reading, so the drawing should say which one the part is judged against.
• The cutoff. The sampling filter changes what the average includes. A stated cutoff makes the number reproducible; an unspecified one leaves it negotiable.
• Where on the feature. Specify the measuring location, or agree that the reading is taken at the functional contact area, away from edges and lead-in marks.

When the finish is functional — a seal face, a bearing journal, a mating surface — it earns this level of specification, because the cost of a wrong reading is a leaking assembly or a failed fit. When the finish is cosmetic, the honest move is the opposite: leave it loose and save the inspection time. The same right-tight discipline that governs tolerances and finishes across the site-wide tolerance guide governs the specification effort here — state the value, the method, and the location on the two or three faces that matter, and leave the rest alone.

What belongs on the inspection report before a tolerance claim is settled?

A dimensional report is only as good as the context printed around the numbers. The measured value against the drawing is the headline, but the paragraphs that make it trustworthy are the instrument used, the reference temperature, the datum structure it was measured from, and the method for any finish or form reading. A report that lists values with no instrument, no temperature, and no datum is a list of opinions; a report that states all three is a measurement.

For finish and form, the method line matters even more, because that is where the reading becomes reproducible or not. A surface finish entry should name the instrument family and cutoff, not just hand you an Ra number. A roundness or position entry should name the CMM and the datum alignment. Material identity should be traceable to the incoming check — an XRF reading against the mill certificate — so the tolerance claim is being judged on the alloy you actually specified.

There is one more line a serious report carries, and it is the one buyers rarely ask for: process evidence, not just part evidence. A single measured part proves that part. A statistical process control chart, with a documented Cpk against the key-characteristic target of Ppk ≥ 1.67 initial, proves the process that made the whole batch was inside its band while the parts were being cut. That distinction — between "we checked one" and "the process was controlled throughout" — is the difference between an inspection result and a capability, and it is the difference a buyer should be paying for.

Which inspection documents should you ask for before volume production?

The paperwork question is best settled before the purchase order, not after the first bad batch. Three documents, requested up front, close most of the verification gap between a quoted tolerance and a delivered one.

• A sample CMM report from a completed job. It should show measured values against the drawing for the tight features, the instrument, and the reference temperature. Ask for it before you commit volume; it costs nothing and tells you what the shop's documentation actually looks like.
• A first-article inspection (FAI) report on your own part. The first piece of a new run is inspected against every characteristic that matters before the batch is built. This is the cheapest place to catch a callout that was ambiguous, unmeasurable, or interpreted differently than you intended.
• SPC evidence for the key characteristics. A control chart with a documented Cpk shows the process stayed centered and inside the band across the run, rather than being sorted after the fact.

Every batch from the facility behind this site ships with a dimensional report covering the critical dimensions, and material is verified by XRF against its mill certificate at receiving. For automotive and regulated programs the same floor produces PPAP-style documentation under IATF 16949:2016. The reason to ask for the documents before production is simple: a supplier that already produces them costs you nothing to verify, and a supplier that does not will not start because you asked nicely.

Which tolerance and finish mistakes repeat on real orders?

The failures below come up often enough on precision work that they are worth treating as a checklist before you release a drawing or sign a PO.

• Over-specifying the finish on a cosmetic surface. A visible face does not need Ra 0.8 µm; it needs to look clean. Finer finishes cost real money — they force grinding or polishing steps — so put the fine number only on the functional contact area.
• Blanket-tolerancing the whole drawing. When every dimension carries ±0.003 mm, the shop has no idea which features actually matter, so it holds and measures everything, and prices it that way. Reserve the tight band for the two or three features that engage another component.
• Ignoring the datum structure. Tight tolerances are unverifiable until they reference a datum the part can be located from in one setup. Without datums, "perpendicular to what" has no answer, and the CMM program has nothing to align to.
• Specifying Ra without a method. As covered above, the value alone is not a testable requirement. Add the instrument family and cutoff, or accept that the number will be argued.
• Trusting a capability sheet over a report. "We hold ±0.003 mm" is a claim; a CMM report and an SPC chart from a completed job are evidence. When a claim and a report disagree, the report wins, and a supplier that cannot produce the report is asking you to buy on faith.
• Measuring at the wrong temperature. A micron-level dimension is only a number at a stated reference temperature, because a warm part reads larger than a cold one. The report should name the reference temperature it was measured at, the drawing should name the temperature the part will be used at, and when those two are far apart, the tightest callout on the print may be unkeepable in service regardless of what the machine produced.

Every one of these mistakes is visible in the documents, not on the surface of the part. That is the recurring theme of this guide: on precision work, the specification and the inspection paperwork are where realism is actually decided, because the part itself will not tell you whether the number on the drawing was ever met.

Frequently Asked Questions

What tolerances are realistic for high precision machining?

Milled and turned features hold ±0.003 mm on a disciplined CNC floor, and precision-ground features hold ±0.002 mm. Those are the working bands this facility runs daily, with critical dimensions verified on Hexagon CMMs and key characteristics charted to Ppk ≥ 1.67 initial.

What surface finishes are realistic, and what does each require?

Ra 3.2 µm is standard milling or turning, Ra 1.6 µm is fine turning or milling for locating faces, Ra 0.8 µm is the precision grinding band for bearing seats and seal faces, and finer finishes come from grinding plus lapping or honing. Specify the coarsest value the function tolerates.

Can I verify a ±0.003 mm tolerance with a caliper or micrometer?

Not reliably. A ±0.003 mm band is only 0.010 mm wide, and hand-tool uncertainty from operator technique and temperature consumes too much of it. A ±0.003 mm claim is a coordinate-metrology claim — it is settled on a CMM, with the reading on a dimensional report.

Why can't I check roundness or flatness with a caliper?

A caliper measures two opposite points and says nothing about the shape between them. Roundness, flatness, and true position are form and location controls that must be measured against a reference — a datum plane or a CMM alignment — so handheld two-point gauges cannot settle them.

Why do two roughness instruments report different Ra for the same part?

Stylus and optical instruments see the same texture through different measurement principles, and Ra depends on the cutoff and evaluation length used. The readings can differ by more than a finishing step without either instrument being wrong, which is why the drawing should state the instrument family and cutoff.

What documentation should ship with a precision order?

A dimensional (CMM) report covering the critical dimensions, material certification cross-checked by XRF at receiving, and SPC charts where the job requires them. For automotive programs, PPAP-style documentation is produced under IATF 16949:2016.

Which quality systems back the tolerance claims on this site?

The certifications behind these parts are third-party audited and current: ISO 9001:2015 with certificate 116024/A/0001/UK/En, IATF 16949:2016 with certificate 131941/A/0001/SM/En, and ISO 14001:2015 with certificate 116024/B/0001/UK/En. The IATF scope covers CNC metal machining as build-to-print, excluding product design under Clause 8.3.

What is the minimum order quantity for tight-tolerance work?

One piece. A single prototype earns its keep by running through the same tolerance and finish verification as a production batch — measured on the CMM, reported against the drawing — which proves the design before volume is committed.