Đáp án:

\(x(1 + 2f(x)) = (f'(x))^2\)

\(\Rightarrow \frac{(f'(x))^2}{1 + 2f(x)} = x \quad \Rightarrow \quad \frac{f'(x)}{\sqrt{1 + 2f(x)}} = \sqrt{x}, \, \forall x \in [1, 4]\)

\(\Rightarrow \int \frac{f'(x)}{\sqrt{1 + 2f(x)}} dx = \int \sqrt{x} dx\)

\(\Rightarrow \sqrt{1 + 2f(x)} = \frac{2}{3} x \sqrt{x} + C\)

\(f(1) = \frac{3}{2} \Rightarrow 2 = \frac{2}{3} \ + C \Rightarrow C = 2 - \frac{2}{3} = \frac{4}{3}\)

\(\Rightarrow 1 + 2f(x) = \left(\frac{2}{3} x \sqrt{x} + \frac{4}{3} \right)^2 = \frac{4}{9}x^3 + \frac{16}{9}x  \sqrt{x} + \frac{16}{9}\)

\(\Rightarrow f(x) = \frac{1}{2}\left[\frac{4}{9}x^3 + \frac{16}{9}x  \sqrt{x}+ \frac{7}{9}\right]\)

\(\Rightarrow \int_{1}^{4} f(x) dx  = \frac{1186}{45} \Rightarrow \boxed{A}\)